Using an eraser over image in Flutter

Question

I'm making a kind-of painting app that has you paint over a background image. There's a couple of tools like pen and then there's eraser. Ideally, we would just paint over the background color, but it's an image. We had the idea of removing the points that were under our finger as we panned across the screen but the offsets and the local position didn't match (they were like 40 (units?) off).

              onPanUpdate: (DragUpdateDetails details) {
                //Adds the point you are tapping on to the '_points' list as Offset(x,y). Later we draw those points.

                if (tool == 'pen') {
                  RenderBox object = context.findRenderObject();
                  Offset _localPosition =
                      object.globalToLocal(details.globalPosition);
                  _points = List.from(_points)..add(_localPosition);
                } else if (tool == 'eraser') {
                  RenderBox object = context.findRenderObject();
                  Offset _localPosition =
                      object.globalToLocal(details.globalPosition);
                  _points = List.from(_points)
                    ..removeWhere(
                      (offset) {
                        if (offset == null) return false;
                        if (offset.dx.round() == _localPosition.dx.round() &&
                            offset.dy.round() == _localPosition.dy.round()) {
                          return true;
                        }
                        return false;
                      },
                    );

                  //This part is not working.
                }
                setState(() {});
              },

If there's another method that I could use, that would be great but I can't see one as of now.