Panning, Zooming and Relative Coordinates in Java Swing

Question

I'm writing a small game in which you can pan (by dragging the mouse) and zoom (by scrolling) and click an image.

The zoom should function as in a Maps-Like Program with the pixel under the cursor staying static and the Image zooming in relation to it (akin to here and here). But those given examples don't do what I want.

The program should also be able to convert Screen-Coordinates (where I clicked) into Image-Relative-Coordinates (which pixel on the image I clicked).

I am using AffineTransform to realize those goals.

This is my current try (a bit simplified to distill my problem):

import javax.imageio.ImageIO;
import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.geom.AffineTransform;
import java.awt.geom.NoninvertibleTransformException;
import java.awt.geom.Point2D;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;

public class ZoomAndPan {
    public static void main(String[] args) throws IOException {
        BufferedImage image = ImageIO.read(new File("path/to/img.png"));

        JFrame frame = new JFrame("Zoom and Pan");
        frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
        frame.setSize(800,600);

        frame.add(new JPanel(){

            private final AffineTransform zoom = new AffineTransform();
            private final AffineTransform pan = new AffineTransform();

            @Override
            protected void paintComponent(Graphics g) {
                super.paintComponent(g);

                Graphics2D g2D = (Graphics2D)g;

                //translate and scale image accoringly
                AffineTransform at = g2D.getTransform();
                at.concatenate(zoom);
                at.concatenate(pan);
                g2D.setTransform(at);

                g2D.drawImage(image,0,0,null);
            }

            {
                MouseAdapter mouseAdapter = new MouseAdapter() {

                    //transform Window-Coords to Image-relative-Coords
                    @Override
                    public void mousePressed(MouseEvent e) {
                        try {
                            Point2D pixelOnImage = zoom.inverseTransform(pan.inverseTransform(e.getPoint(), null), null);
                            if(image.getData().getBounds().contains(pixelOnImage)){
                                System.out.println("You clicked on a Pixel with the color: "+Integer.toHexString(image.getRGB((int) pixelOnImage.getX(), (int) pixelOnImage.getY())));
                            }
                        }catch (NoninvertibleTransformException nTE){
                            System.err.println("This should not happen!");
                        }
                    }

                    //Zooming functionality
                    @Override
                    public void mouseWheelMoved(MouseWheelEvent e) {
                        double zoomFactor = 1-e.getPreciseWheelRotation()/10;
                        zoom.translate(e.getX(),e.getY());
                        zoom.scale(zoomFactor,zoomFactor);
                        zoom.translate(-e.getX(),-e.getY());
                    }


                    //Dragging/Panning functionality
                    private Point dragPoint;
                    @Override
                    public void mouseReleased(MouseEvent e) {
                        dragPoint = null;
                    }
                    @Override
                    public void mouseDragged(MouseEvent e) {
                        if(dragPoint == null) dragPoint = e.getPoint();
                        pan.translate(e.getX()- dragPoint.getX(),e.getY()- dragPoint.getY());
                        dragPoint = e.getPoint();
                    }
                };

                addMouseListener(mouseAdapter);
                addMouseMotionListener(mouseAdapter);
                addMouseWheelListener(mouseAdapter);

                new Timer(5,e -> repaint()).start();
            }
        });

        frame.setVisible(true);
    }
} 

The Panning and Clicking work on their own, but when combined with Zooming, the point to where it zooms is not the cursor and when Clicking while zoomed, I don't get the correct Coordinates/Colors of the image.