Open CV Camera Calibration Projector-like scenario

Question

I have a conceptual understanding problem. Best is to tell you what i have and what i want to achieve: I have two pictures, one distorted, one undistorted (basically both simulated pictures only, can be chessboards, can be anything).

Basically it is not a camera, i am looking at, but a projector, and as i said, i have simulated images only.

My task is to calculate the distortion coefficients with OpenCV.

What i came up so far, failed miserably, basically (i think) because camera calibration works only well with a certain magnitude of images, which are of different planes, so the automatic tutorial-like approach doesn't work.

However, and here i am way out of my league and i am basically just guessing:

I think, i don't really need a Camera matrix, i am superhappy with the Unity Matrix there, i don't need to fiddle around with those coefficients, so i need tvecs, rvecs and the distortion coefficents. I tried camera calibration a lot, the distorted image became a fake object (z=0 [is that correct or just stupid?]), and i get results, but very false results.

tvecs and rvecs alone are not a real problem, i can get those - if i want - with a perspective transform, which works well.

However is there any semiautomatic way how to get the distortion coefficients and only those out of two pictures, one distorted, one undistorted? I tried Camera Calibration without real success

Edit: i wrote some code, maybe this highlights my problem. The manual distortion/undistortion works fine (most parameters zero for simplification), but he cam-calibration is just a horrible mess. Is there any other function i can use and haven't found yet?

#%%

import numpy as np
import cv2
import matplotlib.pyplot as plt

#Barrel distortion effect, which corresponds to negative radial displacement
#Pincushion distortion effect, which corresponds to a positive radial displacement.

# The distortion coefficients
k1 = -3e-2  
k2 = -5e-4 
p1 = 0e-4  
p2 = 0e-2  
k3 = 0e-12
k4 = 0e-5
k5 = 0e-9
k6 = 0e-14
s1 = 0e-3
s2 = 0e-7
s3 = 0e-3
s4 = 0e-7
tx = 0e-1
ty = 0e-3


dist = np.array([k1, k2, p1, p2, k3, k4, k5, k6, s1, s2, s3, s4, tx, ty], dtype = "float32")


# Generate Grid of Object Points
grid_size, square_size = [20, 20], 0.2
object_points = np.zeros([grid_size[0] * grid_size[1], 3], dtype="float32")
mx, my = [(grid_size[0] - 1) * square_size / 2, (grid_size[1] - 1) * square_size / 2]
for i in range(grid_size[0]):
    for j in range(grid_size[1]):
        object_points[i * grid_size[0] + j] = [i * square_size - mx, j * square_size - my, 0]

# Plot the points 
plt.scatter(object_points[:, 0], object_points[:, 1])
plt.title("Original Object, Object Plane")
plt.axis('equal')
plt.xlim((-4, 4))
plt.ylim((-4, 4))
plt.grid()
plt.show()

# Setup the camera information
fx=2.
fy=2.
cx= 0.
cy= 0.
alpha = 0   #### skew factor  accounts for a shear of the coordinate system
mtx = np.array([[fx, alpha, cx], [0, fy, cy], [0, 0, 1]])
rvec = np.array([0.0, 0.0, 0.0])
tvec = np.array([0.0, 0.0, 0.0])


#### Distort the points, res is on the Image Plane
image_points, jacobian = cv2.projectPoints(object_points, rvec, tvec, mtx, dist)

# Plot the points 
plt.scatter(*zip(*image_points[:, 0, :]))
plt.title("distorted, Image Plane")
plt.axis('equal')
plt.xlim((-4, 4))
plt.ylim((-4, 4))
plt.grid()
plt.show()



# undistort Points back with known cameramatrix and Dist-Coefficients, works well
undistpoints= cv2.undistortPoints(src= image_points , cameraMatrix= mtx, distCoeffs=dist)

# Plot the points 
plt.scatter(*zip(*undistpoints[:, 0, :]))
plt.title("after undistort, Object Plane ")
plt.axis('equal')
plt.xlim((-4, 4))
plt.ylim((-4, 4))
plt.grid()
plt.show()

objp =[object_points]
imp= [undistpoints]
retval, cameraMatrix, distCoeffs, rvecs, tvecs = cv2.calibrateCamera(objectPoints=objp, imagePoints=imp, imageSize=(100,100), cameraMatrix= None, distCoeffs= None)

print("Cam Mat:")
print(cameraMatrix)
print("Dist Coeff:")
print(distCoeffs)
print(rvecs)
print(tvecs)