Is there a way I can manipulate the offset scale value in onPinch useGesture in react?

Question

So I want to add zooming button controls to a div that already uses useGesture for the zoom in, zoom out and drag features. The button controls are I am using a code springApi.start({scale: stringStyle.scale.get() - 0.1}) in the onclick function of button. The div does change its scale on button click but as soon as I try to pinch it regains the previous scale and starts from there. The flinching does not looks good. Any suggesstion or any other library that i can use for this purpose would be helpful. Below is my code-

useGesture({
  onDrag: ({ pinching, cancel, offset: [x, y], ...rest }) => {
    if (pinching) return cancel();
    springApi.start({ x, y });
  },
  onPinch: ({ origin: [ox, oy], first, movement: [ms], offset: [s, a], memo }) => {
    if (first) {
      const { width, height, x, y } = ref.current!.getBoundingClientRect();
      const tx = ox - (x + width / 2);
      const ty = oy - (y + height / 2);
      memo = [springStyle.x.get(), springStyle.y.get(), tx, ty];
    }
    const x = memo[0] - (ms - 1) * memo[2];
    const y = memo[1] - (ms - 1) * memo[3];
    springApi.start({ scale: s });
    return memo;
  },
});

I have tried the react-pan-pinch-zoom library but it is very slow on pinch gesture and I also wanted to add onWheel gesture so I thought useGesture will be better. please help