Unnecessary rebuild of a parent widget upon first detected click on LayoutBuilder/Canvas in Flutter

Question

I am currently writing an application where I need to draw a bunch of shapes using CustomPainter and LayoutBuilder on the screen.

My CustomPainter is basically wrapped with a stateful widget in order to change the color and any other states after clicking. The information regarding all these individual drawings on canvas is kept in a Notifier from Riverpod.

Then whenever a shape is clicked, I update the color using

shape.key.currentState!.updateColor();

I am detecting the particular shape that is being clicked using LayoutBuilder. (Hence, using the coordinates of the click + coordinates of the shapes I have drawn.

However, there is one singular problem. I do not want the entire app to be rebuilt when updating a singular shape. But for some reason, the entire app gets rebuilt after the first click. It is fine for the clicks coming after that, however, the first click always rebuilds the entire app. What could be the reason for that. I will attach the necessary code snippets down.

This is the main widget that I am rendering.

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    print('reubilding the app');
    return ProviderScope(
      child: MaterialApp(
        home: SafeArea(
          child: Scaffold(
            body: Consumer(
              builder: (_, WidgetRef ref, __) {
                final grid = ref.watch(gridStateProvider);
                return Container(
                  color: Colors.white,
                  padding: const EdgeInsets.all(8),
                  child: LayoutBuilder(
                    builder: (context, constraints) {
                      grid.initialize(
                        constraints.maxWidth,
                        constraints.maxHeight,
                      );

                      return Listener(
                        onPointerDown: (PointerEvent details) {
                          ref
                              .read(gridStateProvider.notifier)
                              .handleClick(details);
                        },
                        child: Container(
                          width: constraints.maxWidth,
                          height: constraints.maxHeight,
                          color: Colors.transparent,
                          child: Stack(
                            children: grid.shapes,
                          ),
                        ),
                      );
                    },
                  ),
                );
              },
            ),
          ),
        ),
      ),
    );
  }
}

This is the start of the custom painter object with the methods I am using update each shape individually.

class HexagonPaint extends StatefulWidget {
  final HexagonModel model;

  @override
  final GlobalKey<_HexagonPaintState> key = GlobalKey<_HexagonPaintState>();
  HexagonPaint(this.model, {super.key});

  @override
  _HexagonPaintState createState() => _HexagonPaintState();
}

class _HexagonPaintState extends State<HexagonPaint> {
  pressed() {
    setState(() {
      widget.model.pressed = !widget.model.pressed;
    });
  }