g2o: How to optimize camera intrinsic (fx,fy,cx,cy) during Bundle Adjustment

Question

Some pre-defined edges, such as EdgeProjectXYZ2UV, EdgeSE3ProjectXYZ, are widely used during BA. However, they can only set unchangeable camera parameters. I'm wondering if I can set optimizable camera parameters.

Thanks for any reply in advance!

I found Vertex VertexIntrinsics in g2o/types/sba/vertex_intrinsics.h. And its oplusImpl() function is implemented which means it can be optimized.

/**
 * \brief Vertex encoding the intrinsics of the camera fx, fy, cx, xy, baseline;
 */
class G2O_TYPES_SBA_API VertexIntrinsics : public BaseVertex<4, Eigen::Matrix<number_t, 5, 1, Eigen::ColMajor> >

It seems to be the one I'm looking for. Yet I cannot find a type of Ternary-Edge that link this vertex, camera pose vertex, and 3D world point vertex. Or any other edge that takes this type of vertex.

It would be great if somebody could share an example or some explanation about how this vertex works. Or any other way to optimize camera intrinsic in g2o.

Answer 1

I managed to solve it by defining a Ternary edge that links the camera intrinsic Vertex (VertexIntrinsics), camera pose Vertex (VertexSE3 or VertexSE3Expmap), and Observation Target (Vertex SE3 in my case, or VertexPointXYZ ). There will be only one VertexIntrinsics in the graph.

If you are looking for a ternary edge example, check edge_project_psi2uv.h.

Thank @RainerKuemmerle for such great work.