Inverse Rotation and Translation for Pinhole Projection

Question

So, I feel really dumb to try to solve this problem for so long. I really need help.

I am trying to project the camera pixel coordinate to the world coordinate using Pinhole perspective projection. But somehow the rays I generate has weird rotation (direction). So the function is:

def pixel_coords_to_world_coords_ray(pixel_coords, camera_matrix, distortion_coeffs, R_world_to_camera, t_world_to_camera, plane_normal_world, plane_distance_world):
    
    pixel_coords_undistorted, newcameramtx = undistort_contour_pixel(pixel_coords, camera_matrix, distortion_coeffs)
    pixel_coords_undistorted = np.squeeze(pixel_coords_undistorted, axis=1)
    pixel_coords_homogeneous = np.hstack((pixel_coords_undistorted, np.ones((len(pixel_coords_undistorted), 1))))
    
    # Invert the camera matrix to get pixel to ray conversion matrix
    camera_matrix_inv = np.linalg.inv(newcameramtx)
    
    # Compute the light ray direction in the camera frame.
    ray_directions_camera = (camera_matrix_inv @ pixel_coords_homogeneous.T)[:3, :]

    # Transform the light ray direction to the world frame
    R_camera_to_world = R_world_to_camera.T
    t_camera_to_world = -R_camera_to_world @ t_world_to_camera
    ray_directions_world = R_camera_to_world @ ray_directions_camera
    
    # Normalize the rays to have unit length
    ray_directions_world /= np.linalg.norm(ray_directions_world, axis=0)
    
    # Compute the intersection between the light rays and the plane
    ray_origin_world = t_camera_to_world.reshape(3)
    world_coords = []
    for ray_direction_world in ray_directions_world.T:
        intersection = ray_plane_intersection(ray_origin_world, ray_direction_world, plane_normal_world, plane_distance_world)
        if intersection is not None:
            world_coords.append(intersection)
    world_coords = np.array(world_coords)

    return world_coords

• So I've tried to debug it by setting the input pixel to be (w/2, h/2). It results to the expected value of ray_directions_camera = near [0,0,1]

But somehow the rotation I get in the ray's direction in world frame is wrong.

• I also plot the result, the transformation and rotation of the camera. So the translation and rotation must be right.

the red line pointing downward at the middle is the resulting ray for the pixel (w/2, h/2)

Am I missing something here? It's been a week I am trying to solve this with many different method. I really need to finish this. Thank you all.