undistort vs. undistortPoints for feature matching of calibrated images

Question

I am trying to find the euclidean transform between two cameras (or actually, one moving camera) capturing the same scene, where calibration data K (intrinsic params) and d (distortion coefficients) is known. I am doing this via extraction of feature points, matching them and using the best matches as correspondences.

Prior to resizing/feature detection/etc. I undistort both images

undistort(img_1, img_1_undist, K, d);
undistort(img_2, img_2_undist, K, d);

where img_. are the inputs in Mat form obtained by imread. But really I only need the undistorted coordinates of the features I eventually use as correspondences, not those of all image pixels, so it would be more efficient, not to undistort the whole images, but only the keypoints. I thought I could do just that with undistortPoints, however the two approaches lead to different results.

I resize the images

 resize(img_1_undist, img_1_undist, Size(img_1_undist.cols / resize_factor,
            img_1_undist.rows / args.resize_factor));
 resize(img_2_undist, img_2_undist, Size(img_2_undist.cols / resize_factor,
            img_2_undist.rows / args.resize_factor));
 // scale matrix down according to changed resolution
 camera_matrix = camera_matrix / resize_factor;
 camera_matrix.at<double>(2,2) = 1;

After obtaining best matches in matches, I build std::vectors for the coordinates of said matches,

    // Convert correspondences to vectors
    vector<Point2f>imgpts1,imgpts2;
    cout << "Number of matches " << matches.size() << endl;
    for(unsigned int i = 0; i < matches.size(); i++)
    {
       imgpts1.push_back(KeyPoints_1[matches[i].queryIdx].pt);
       imgpts2.push_back(KeyPoints_2[matches[i].trainIdx].pt);
    }

Which I then use for finding the essential matrix.

    Mat mask; // inlier mask
    vector<Point2f> imgpts1_undist, imgpts2_undist;
    imgpts1_undist = imgpts1;
    imgpts2_undist = imgpts2;
    /* undistortPoints(imgpts1, imgpts1_undist, camera_matrix, dist_coefficients,Mat(),camera_matrix); // this doesn't work */
    /* undistortPoints(imgpts2, imgpts2_undist, camera_matrix, dist_coefficients,Mat(),camera_matrix); */
    Mat E = findEssentialMat(imgpts1_undist, imgpts2_undist, 1, Point2d(0,0), RANSAC, 0.999, 8, mask);

When I remove the calls to undistort and instead call undistortPoints on the keypoints, it does not produce the same result (which I would expect). The differences are sometimes minor, but always there.

I read the documentation

The function is similar to cv::undistort and cv::initUndistortRectifyMap but it operates on a sparse set of points instead of a raster image.

such that the function should do what I expect. What am I doing wrong?

Answer 1

You are seeing the discrepancy because undistorting an image and undistorting a set of points works very differently.

Images are undistorted using inverse mapping, which is the same method that is typically used for all geometric image transformations, such as rotation. You first create the output image grid, and then transform each pixel in the output image back into the input image, and get the value by interpolation.

Since your output image contains the "correct" points, you have to "distort" them to transform them into the original image. In other words, you simply apply the distortion equations.

On the other hand, if you take points from your input image, and try to remove distortion, you would need to invert the distortion equations. That is very hard to do, because those equations are 4th or 6th degree polynomials. So undistortPoints does that numerically, using gradient descent, which will have some error.

To summarize: the undistort function undistorts the entire image, which may be an overkill, but it does that fairly accurately. If you are only interested in a small set of points, undistortPoints is likely to be faster, but it is also likely to have a higher error.