Display text on point cloud vertex in pyrender/open3d

Question

I am using SMPL demo code to display the SMPL-X mesh of a person. Instead of taking the default body joints, I need to select vertices corresponding to some other positions on the mesh, and for that I need to figure out the respective indices of those joints. Easiest way of doing that that I could think of is simply displaying some text over each vertex with its numeric id, will look messy but zooming in should be a fast way of getting what I need.

The demo code I linked above is using 3 separate libraries to display the mesh: pyrender, matplotlib and open3d. Matplotlib is super slow and becomes unusable when displaying all the vertices in the mesh, but it's also the only one where it's trivial to display the vertex indices since it's as easy as:

    for idx, j in enumerate(joints):
        ax.text(j[0], j[1], j[2], str(idx), None)

And produces this

which is exactly what I need. Looks messy but zooming in makes it much more readable, except it takes like 5 minutes to get to the right position on the hands because of how terrible matplotlib 3D visualization is.

I have spent 1 hour trying to figure out how to achieve the same using either pyrender or open3d, but I'm not familiar with neither of the two and from what I could find, there isn't a straightforward way of putting this kind of text on some positions in a point cloud. Does anyone have an idea on how to do this with either one of these libs? Any help will be much appreciated!

Answer 1

I'm currently working with Open3D so I might have some insights.

I suggest reading the Open3D docs about io.

You can easily load a mesh with open3d.io.read_triangle_mesh('path/to/mesh') and add a Label with sceneWidget.add_3d_label([x, y, z], 'label text').

Below is an small example. If you need mouse-events it's quite a bit more complicated but for simply displaying it this might work. Note that I have only tested it with a point cloud.

import open3d as o3d
import open3d.visualization.gui as gui
import open3d.visualization.rendering as rendering

if __name__ == "__main__":
    gui.Application.instance.initialize()

    window = gui.Application.instance.create_window("Mesh-Viewer", 1024, 750)

    scene = gui.SceneWidget()
    scene.scene = rendering.Open3DScene(window.renderer)

    window.add_child(scene)

    # mesh = o3d.io.read_triangle_mesh('path/to/data', print_progress=True)
    mesh = o3d.io.read_point_cloud('path/to/data', print_progress=True)

    scene.scene.add_geometry("mesh_name", mesh, rendering.MaterialRecord())

    bounds = mesh.get_axis_aligned_bounding_box()
    scene.setup_camera(60, bounds, bounds.get_center())

    labels = [[0, 0, 0]]
    for coordinate in labels:
        scene.add_3d_label(coordinate, "label at origin")

    gui.Application.instance.run()  # Run until user closes window