2D pixel coordinates to 3D world coordinates using opencv solvePnP

Question

I saw a lot of similar questions, but not this exact one.

I know:

• pixel coordinates (u,v) and depth (d) - 4 points
• corresponding world coordinates (x,y,z) - 4 points
• intrinsic matrix K of the camera (I didn't calibrate yet but used the camera default one)
• extrinsic matrix [R|t] (I call its 4x4 version M_ext)
• conversion from world coordinate points X (3D) to pixel coordinate points p (2D)

I don't know:

• conversion from pixel coordinate points p (2D) + depth d to world coordinate points X (3D)

0. Get extrinsic matrix:

To get the extrinsic matrix I use opencvs function solvePnP:

ret, rvec, tvec = cv.solvePnP(world_points, img_points, K, np.float32([]), cv.SOLVEPNP_IPPE)

I get the extrinsic matrix [R|t] using the rotation and translation vector above ( Rt = np.hstack((cv.Rodrigues(rvec)[0], tvec))).

I want to apply both conversions for validation that my extrinsic matrix is correct.

1. World coordinate points to pixel coordinates:

camera_coordinates = K * M_ext * world_coordinates 
u = camera_coordinates[0] / camera_coordinates[2]
v = camera_coordinates[1] / camera_coordinates[2]

This conversion works, my pixel coordinates u, v are correct.

2. Pixel coordinates + depth to world coordinate points:

camera_coordinates = ( inv(K) * [u, v, 1] ) * d
world_coordinates = inv(M_ext) * [camera_coordinates, 1]

This inverse conversion does not yield my world coordinate points. Does anyone see where my error is?

Answer 1

first, sorry for before . then i need say i was in the same situation as you, and the problem had been solved already. i tried several different strategies, the following 2 solved my problem:

strategy #1

1. kicked out a few images from my calibration images set.
2. calibrated my camera again.
3. applied the inverse conversion with new parameters.

problem solved!

strategy #2

1. took a few more cal-tab images and added to images set.
2. calibrated my camera again.
3. applied the inverse conversion with new parameters.

problem solved!

i think the problem is caused by some outlier data when try to calibrate camera. but i didn't read the source code of opencv , so i don't know in detail, . hope my experience can help.