OpenCV Stereo Calibration Rotation Matrix

Question

I'm trying to undistort the output of a pair of cameras using the following OpenCV functions:

cv::stereoCalibrate(object_points, image_points_left, image_points_right,cameraMatrixLeft, distCoeffsLeft, cameraMatrixRight, distCoeffsRight,cv::Size(sensor_width, sensor_height),R, T, E, F);

cv::stereoRectify(cameraMatrixLeft, distCoeffsLeft, cameraMatrixRight, distCoeffsRight, cv::Size(sensor_width, sensor_height), R, T, R1, R2, P1, P2, Q);

cv::undistortPoints(src_left, dst_left, cameraMatrixLeft, distCoeffsLeft, R1, P1);
cv::undistortPoints(src_right, dst_right, cameraMatrixRight, distCoeffsRight, R2, P2);

As I understand, the function cv::undistortPoints should also rectify the output using the matrices R1, R2, P1 and P2 according to my stereo camera setup, i.e. set the epipolar lines parallel to the image y-axis and crop the image.

Is this correct?

However, the mapping performed on the points in src_left and src_right is "tilted", i.e. the undistorted output images are tilted, leaving blank spaces on the boarders (especially in half of the corners) of the image. I believe this tilting comes from the rotation matrices R1 and R2 and corresponds to the rotations around the z-axis.

Now here is my question:

• Why do the (rectified) output images not fill the whole image space?

Answer 1

During rectification and undistortion the image can be translated and/or rotated depending on orientation of the stereo cameras.

Obviously, if you will leave the original resolution for the transformed image - you will lose some information around the border. E.g. consider that there's no translation, but you have to rotate one of the images 45 degrees clockwise and use the same resolution - the corners will be "blank".

The common solution is to find a largest "inscribed" rectangle and scale it up to the original resolution.

You can use initUndistortRectifyMap for that instead of undistortPoints.

For your stereoRectify call, the calls to the initUndistortRectifyMap will look as follows:

• left: initUndistortRectifyMap(cameraMatrixLeft, distCoeffsLeft, R1, P1, cv::Size(sensor_width, sensor_height), CV_32FC1, map_left_1, map_left_2);
• right: initUndistortRectifyMap(cameraMatrixRight, distCoeffsRight, R2, P2, cv::Size(sensor_width, sensor_height), CV_32FC1, map_right_1, map_right_2);

This will get you the maps for a call to remap:

• remap(left, left_rectified, map_left_1, map_left_2, INTER_LINEAR, BORDER_CONSTANT);
• remap(right, right_rectified, map_right_1, map_right_2, INTER_LINEAR, BORDER_CONSTANT);