Fish-eye warping about mouse position - fragment shader

Question

I'm trying to create a fish-eye effect but only in a small radius around the mouse position. I've been able to modify this code to work about the mouse position (demo) but I can't figure out where the zooming is coming from. I'm expecting the output to warp the image similarly to this (ignore the color inversion for the sake of this question):

Relevant code:

// Check if within given radius of the mouse
vec2 diff = myUV - u_mouse - 0.5;
float distance = dot(diff, diff); // square of distance, saves a square-root

// Add fish-eye 
if(distance <= u_radius_squared) {
  vec2 xy = 2.0 * (myUV - u_mouse) - 1.0;
  float d = length(xy * maxFactor);
  float z = sqrt(1.0 - d * d);
  float r = atan(d, z) / PI;
  float phi = atan(xy.y, xy.x);
    
  myUV.x = d * r * cos(phi) + 0.5 + u_mouse.x;
  myUV.y = d * r * sin(phi) + 0.5 + u_mouse.y;
}

vec3 tex = texture2D(tMap, myUV).rgb;

gl_FragColor.rgb = tex;

This is my first shader, so other improvements besides fixing this issue are also welcome.

Answer 1

Compute the vector from the current fragment to the mouse and the length of the vector:

vec2 diff = myUV - u_mouse;
float distance = length(diff);

The new texture coordinate is the sum of the mouse position and the scaled direction vector:

myUV = u_mouse + normalize(diff) * u_radius * f(distance/u_radius);

For instance:

uniform float u_radius;
uniform vec2 u_mouse;

void main()
{
    vec2 diff = myUV - u_mouse;
    float distance = length(diff);

    if (distance <= u_radius)
    {
        float scale = (1.0 - cos(distance/u_radius * PI * 0.5));
        myUV = u_mouse + normalize(diff) * u_radius * scale;
    }

    vec3 tex = texture2D(tMap, myUV).rgb;
    gl_FragColor = vec4(tex, 1.0);
}