Mapping 3d vertex to pixel using pyreder/pyglet/openGL

Question

I'm implementing new algorithm for 3d point estimation using images, and right now I'm trying to test it over 3d virtual models before I'll move to real objects.

The algorithm inputs are pixels before the last transformation to the viewport sizes, so to test the algorithm on rendered images, I need to know the reverse transformation from pixel in the shape([0,witdh],[0,height]).

I'm using perspective projection from the library pyrender to render the 2d images of 3d mesh, as far as I know this library using OpenGL methods for rendering.

example: I have a mesh of a box with the sizes=(3,1,5), center=(0,0,0) , and I have the Projection and View matrices

View Matrix=       [[ 0.96592583, -0.0669873 ,  0.25     ,  3.        ],
                   [ 0.        ,  0.96592583,  0.25881905,  4.        ],
                   [-0.25881905, -0.25      ,  0.9330127 , 10.        ],
                   [ 0.        ,  0.        ,  0.        ,  1.        ]]

Projection Matrix= [[ 2.4142135,  0.        ,  0.        ,  0.        ],
                   [ 0.        ,  2.41421356,  0.        ,  0.        ],
                   [ 0.        ,  0.        , -1.0010005 , -0.10005003],
                   [ 0.        ,  0.        , -1.        ,  0.        ]]

this is my calculation to map 3d point/vertex into pixel:

def map_to_pixel(point3d,w,h,projection,view):
    p=projection@view@point3d # openGL perspective projection
    p=p/p[3]                  # divide by the w element
    p[0]=w/2*p[0]+w/2         # transformation from [-1,1] -> [0,width]
    p[1]=h/2*p[1]+h/2         # transformation from [-1,1] -> [0,height]
    return p

test it on the vertex top-left-close of the box =[-1.5, 0.5, 2.5, 1. ] when viewport_sizes=(width,height)=(512,512)

results= [150.86775635, 4.28475523, 1.00894365, 1. ]= (151,4)

when the actual results by pyrender for this vertex is the pixel ~ (90,342)

if anyone knows the actual process behind the scenes of pyrender/OpenGL, or knows how to map the pixels correctly, that will be super helpful.

btw: I know my function uses bottom-left mapping when the library uses top-left mapping, but it still gives unexpected output.

Answer 1

I figured out how goes the calculation, I don't know why, but the library pyrender (that uses OpenGL methods), use the inverse matrix of the view matrix I set as an input.

the exact function to set a pixel is:

from numpy.linalg import inv

    def map_to_pixel(point3d,w,h,projection,view):
        p=projection@inv(view)@point3d.T
        p=p/p[3]
        p[0]=(w/2*p[0]+w/2)    #tranformation from [-1,1] ->[0,width]
        p[1]=h-(h/2*p[1]+h/2)  #tranformation from [-1,1] ->[0,height] (top-left image)
        return p

I have tested the rendered images across this function, and all the vertices have been mapped perfectly into the pixels.