How to combine non linear methods of PnP, ICP and direct method?

Question

I'm currently working on some visual odometry algorithms, I found out that there are non linear and linear methods in solving problems such as PnP and ICP, while the direct methods minimizes the intensity errors between two points. The nonlinear methods above also minimizes errors between two points on reprojection errors.

I can implement PnP, ICP and direct methods solving with Gauss-Newton methods, as their implementation only differs in the jacobian matrix and the error function, I wanna ask is there any way to minimize three error functions and combined it together ? For example, the opencv library has combined ICP and direct method in "opencv_contrib/odometry.cpp" as I paste it below, but I can't quite understand the principle in the "calcRgbdLsmMatrices" function, is there other ways to combine three of them using non-linear method?

Thanks for any help in advance !!

for(int iter = 0; iter < iterCounts[level]; iter ++) 
        {
            Mat resultRt_inv = resultRt.inv(DECOMP_SVD);
            if(method & RGBD_ODOMETRY)
                computeCorresps(levelCameraMatrix, levelCameraMatrix_inv, resultRt_inv,
                                srcLevelDepth, srcFrame->pyramidMask[level], dstLevelDepth, dstFrame->pyramidTexturedMask[level],
                                maxDepthDiff, corresps_rgbd);

            if(method & ICP_ODOMETRY)
                computeCorresps(levelCameraMatrix, levelCameraMatrix_inv, resultRt_inv,
                                srcLevelDepth, srcFrame->pyramidMask[level], dstLevelDepth, dstFrame->pyramidNormalsMask[level],
                                maxDepthDiff, corresps_icp);
            if(corresps_rgbd.rows < minCorrespsCount && corresps_icp.rows < minCorrespsCount)
                break;

            Mat AtA(transformDim, transformDim, CV_64FC1, Scalar(0)), AtB(transformDim, 1, CV_64FC1, Scalar(0));
            if(corresps_rgbd.rows >= minCorrespsCount) // 120
            {
                calcRgbdLsmMatrices(srcFrame->pyramidImage[level], srcFrame->pyramidCloud[level], resultRt,
                                    dstFrame->pyramidImage[level], dstFrame->pyramid_dI_dx[level], dstFrame->pyramid_dI_dy[level],
                                    corresps_rgbd, fx, fy, sobelScale,
                                    AtA_rgbd, AtB_rgbd, rgbdEquationFuncPtr, transformDim);

                AtA += AtA_rgbd;
                AtB += AtB_rgbd;
            }
            if(corresps_icp.rows >= minCorrespsCount)
            {
                calcICPLsmMatrices(srcFrame->pyramidCloud[level], resultRt,
                                   dstFrame->pyramidCloud[level], dstFrame->pyramidNormals[level],
                                   corresps_icp, AtA_icp, AtB_icp, icpEquationFuncPtr, transformDim);
                AtA += AtA_icp;
                AtB += AtB_icp;
            }