OpenCV to OpenGL perspective projection matrix slight error

Question

I have been struggling with trying to get the perspective projection matrix for opengl correctly. Using opencv, i am able to overlay a cube(red wiremesh) over the marker perfectly. However, when i apply the opencv-opengl perspective projection from the post (http://www.morethantechnical.com/2015/02/17/augmented-reality-on-libqglviewer-and-opencv-opengl-tips-wcode/) as follows, a solid cube is overlayed on the marker, however when i rotate the marker, the solid cube slowly shifts out of the marker (https://www.youtube.com/watch?v=m87qAZcN6c4&feature=youtu.be).The red cube (wiremesh) was drawn using opencv and it is overlayed perfectly.

// perspective projection matrix*************************
double fx = intrinsic_matrix.at<double>(0,0);
double fy = intrinsic_matrix.at<double>(1,1);
double cx = intrinsic_matrix.at<double>(0,2);
double cy = intrinsic_matrix.at<double>(1,2);
projection.at<double>(0,0) = fx/cx;
projection.at<double>(1,1) = fy/cy;
projection.at<double>(2,2) = -(far+near)/(far-near);
projection.at<double>(2,3) = -2.0*far*near / (far-near);
projection.at<double>(3,2) = -1.0;

projection = projection.t();

// modelview matrix **********************************************
modelview.at<double>(0,0) = rotation.at<double>(0,0);
modelview.at<double>(1,0) = rotation.at<double>(1,0);
modelview.at<double>(2,0) = rotation.at<double>(2,0);
modelview.at<double>(3,0) = 0;

modelview.at<double>(0,1) = rotation.at<double>(0,1);
modelview.at<double>(1,1) = rotation.at<double>(1,1);
modelview.at<double>(2,1) = rotation.at<double>(2,1);
modelview.at<double>(3,1) = 0;

modelview.at<double>(0,2) = rotation.at<double>(0,2);
modelview.at<double>(1,2) = rotation.at<double>(1,2);
modelview.at<double>(2,2) = rotation.at<double>(2,2);
modelview.at<double>(3,2) = 0;

modelview.at<double>(0,3) = tvecs.at<double>(0);
modelview.at<double>(1,3) = tvecs.at<double>(1);
modelview.at<double>(2,3) = tvecs.at<double>(2);
modelview.at<double>(3,3) = 1;

// This matrix corresponds to the change of coordinate systems.
static double changeCoordArray[4][4] = {{1, 0, 0, 0}, {0, -1, 0, 0}, {0, 0, -1, 0}, {0, 0, 0, 1}};
static Mat changeCoord(4, 4, CV_64FC1, changeCoordArray);

modelview = changeCoord*modelview;
modelview = modelview.t();

Can anybody help me fix this? Here is my code (http://pastebin.com/2MkQvkdN).

Answer 1

I think the cause of this is that you are comparing each frame with the previous frame to determine where the solid cube should move to. If this is the case, I would try comparing each frame with the first frame, to determine how the cube should move. This should get rid of this numerical drift. Alternatively, you could try placing the solid cube for each individual frame, based on where you detect the black and white markers.