OpenCV stereo camera calibration: support mixed camera models (normal + fisheye)?

Question

In OpenCV APIs, there are cv::stereoCalibrate() and cv::fisheye::stereoCalibrate() for calibrating normal stereo cameras and fisheye stereo cameras repectively. For both APIs, the stereo camera pair use the same camera model. In other words, both stereo cameras must use normal camera model (6 radial distortion coefficients + 2 tangential ones) for cv::stereoCalibrate() or fisheye camera model (4 fisheye distortion coefficients) for cv::fisheye::stereoCalibrate().

Is there any way to calibrate a stereo camera pair where one use normal camera model and the other use fisheye camera model via OpenCV?

Answer 1

For anyone find this question helpful, OpenCV doesn't natively support stereo calibration (i.e., via stereoCalibrate() API) for mixed camera models (e.g., one is normal, and the other fisheye). However, this can be done by the following steps:

1. Calibrate each camera's intrinsics via API cv::calibrateCamera() or cv::fisheye::calibrate().
2. Undistort all image points for both cameras use previous intrinsics via API cv::undistortPoints() or cv::fisheye::undistortPoints().
3. Calibrate the extrinsics for the stereo pair using the image points after undistortion in previous step and "perfect intrinsics" for both cameras via API cv::stereoCalibrate() or cv::fisheye::stereoCalibrate() (with CALIB_FIX_INTRINSIC flag to compute extrinsics only). A camera with perfect intrinsics means its camera matrix is [1 0 0; 0 1 0; 0 0 1] and has zero distortion (distortion coefficients are all 0).

   Note: It doesn't matter to use cv::stereoCalibrate() or cv::fisheye::stereoCalibrate() in this step, which will result in the same extrinsics.