I have a stereopair (2 perspective cameras) which is rotated and does not move. Using conventional opencv methods I found disparity maps and perspective projection matrix Q.
From this I constructed 2 point clouds with reprojectImageTo3D() for two different stereopair orientations. I do know the angles describing camera orientation for each cloud (disparity map).
My question is how to transform (rotate, translate) those clouds to get a match? What math is behind?
I thought naively that it would be sufficient just to do the rotation of the second cloud by inverse rotation angles, but it does not work both from practical and theoretical point of view. Any ideas?