How to show an image in the background of a surface plot in vispy?

Question

I want to have a surface plot in vispy, but also display a 2D image in place of the white background.

I used vispy's Draw a SurfacePlot example as a base. The first thing I tried is just adding a scene.visuals.Image object to the view, but that ended up rendering the image as one of the objects in the 3D scene:

canvas = scene.SceneCanvas(keys='interactive', bgcolor='w')
background_texture = scene.visuals.Image(np.random.rand(512, 512, 3).astype(np.float32))

view = canvas.central_widget.add_view()
view.camera = scene.TurntableCamera(up='z', fov=60)
view.add(background_texture)

...

I don't have reputation to display images: failed image background surfaceplot attempt #1

Since the image needs to be rendered in the background, I figured it needs its own 2D view on top of which the 3D scene of the SurfacePlot will be rendered, so I created a new view for the image:

canvas = scene.SceneCanvas(keys='interactive', bgcolor='w')
background_texture = scene.visuals.Image(np.random.rand(512, 512, 3).astype(np.float32))

viewbg = canvas.central_widget.add_view()
viewbg.add(background_texture)

view = canvas.central_widget.add_view()
view.camera = scene.TurntableCamera(up='z', fov=60)

...

However, the image is rendered on top of the surface plot, with the exception of axis labels which are rendered on top of the image. Reordering view and viewbg code so view is created first doesn't make a difference; it only makes things worse because then I also lose the ability to interact with the surface plot.

I don't have reputation to display images: failed image background surfaceplot attempt #2

Is there a way to render the entire 3D scene on top of the image?

Answer 1

Thanks to guidance from @djhoese, I was able to come up with some solutions. First and foremost, the background image must be outside of view in the scene graph, lest it becomes a part of the 3D space displayed by the view's TurntableCamera. After that comes the problem of actually displaying the image behind the 3D scene.

Solution 1: Clear the OpenGL depth buffer after drawing the image and before drawing the 3D scene

The cleanest approach would be to first draw the image, then clear the OpenGL depth buffer before drawing the 3D scene. This will ensure that the scene is drawn on top of the image. The easiest way to do this would be to subclass scene.Image and add a call to gloo.clear(color=False, depth=True) after drawing:

class BackgroundImage(scene.Image):
    def draw(self):
        super().draw()
        gloo.clear(color=False, depth=True)

Ensure that BackgroundImage is rendered before anything else. Full code below:

import sys
import numpy as np

from vispy import app, scene, gloo
from vispy.util.filter import gaussian_filter


class BackgroundImage(scene.Image):
    def draw(self):
        super().draw()
        gloo.clear(color=False, depth=True)


canvas = scene.SceneCanvas(keys='interactive', bgcolor='w')

view_bg = canvas.central_widget.add_view(camera=scene.PanZoomCamera())
view_bg.order = float("-inf")  # For good measure

background_image = BackgroundImage(np.random.rand(512, 512, 3).astype(np.float32), parent=view_bg.scene)
view_bg.camera.set_range(margin=0)

view = canvas.central_widget.add_view()
view.camera = scene.TurntableCamera(up='z', fov=60)

# Simple surface plot example
# x, y values are not specified, so assumed to be 0:50
z = np.random.normal(size=(250, 250), scale=200)
z[100, 100] += 50000
z = gaussian_filter(z, (10, 10))
p1 = scene.visuals.SurfacePlot(z=z, color=(0.3, 0.3, 1, 1))
p1.transform = scene.transforms.MatrixTransform()
p1.transform.scale([1/249., 1/249., 1/249.])
p1.transform.translate([-0.5, -0.5, 0])

view.add(p1)

# p1._update_data()  # cheating.
# cf = scene.filters.ZColormapFilter('fire', zrange=(z.max(), z.min()))
# p1.attach(cf)


xax = scene.Axis(pos=[[-0.5, -0.5], [0.5, -0.5]], tick_direction=(0, -1),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
xax.transform = scene.STTransform(translate=(0, 0, -0.2))

yax = scene.Axis(pos=[[-0.5, -0.5], [-0.5, 0.5]], tick_direction=(-1, 0),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
yax.transform = scene.STTransform(translate=(0, 0, -0.2))

# Add a 3D axis to keep us oriented
axis = scene.visuals.XYZAxis(parent=view.scene)

if __name__ == '__main__':
    canvas.show()
    if sys.flags.interactive == 0:
        app.run()

A PanZoomCamera is used with background_image to ensure that the image is scaled to fit the entire canvas. You won't be able to interact with that camera since the view with the TurntableCamera is drawn on top so it takes focus.

I don't have reputation to embed images in my answer: surface plot with a background image

Generalizing the solution to draw any scene in the background

If you want to draw any scene, not just an Image in the background of another scene, you can implement a Node subclass that clears the depth buffer after its children are drawn. It's probably best to inherit from Widget since it provides the most features:

class Background(scene.Widget):
    """Node that clears the depth buffer after drawing its children."""
    ClearDepth = type("ClearDepthBuffer", (scene.Node,), {'draw': lambda self: gloo.clear(color=False, depth=True)})

    def __init__(self, *args, **kwargs):
        self._last = None
        super().__init__(*args, **kwargs)
        self._last = self.ClearDepth(parent=self)
        self._last.order = float("inf")
        assert self._children.pop() == self._last

    @property
    def children(self):
        return super().children + [self._last] if self._last is not None else super().children

Make sure that Background is actually drawn before anything else. Full surface plot code with this approach below:

import sys
import numpy as np

from vispy import app, scene, gloo
from vispy.util.filter import gaussian_filter


class Background(scene.Widget):
    """Node that clears the depth buffer after drawing its children."""
    ClearDepth = type("ClearDepthBuffer", (scene.Node,), {'draw': lambda self: gloo.clear(color=False, depth=True)})

    def __init__(self, *args, **kwargs):
        self._last = None
        super().__init__(*args, **kwargs)
        self._last = self.ClearDepth(parent=self)
        self._last.order = float("inf")
        assert self._children.pop() == self._last

    @property
    def children(self):
        return super().children + [self._last] if self._last is not None else super().children


canvas = scene.SceneCanvas(keys='interactive', bgcolor='w')

background = canvas.central_widget.add_widget(Background())
background.order = float("-inf")  # For good measure
view_bg = background.add_view(camera=scene.PanZoomCamera())

background_image = scene.Image(np.random.rand(512, 512, 3).astype(np.float32), parent=view_bg.scene)
view_bg.camera.set_range(margin=0)

view = canvas.central_widget.add_view()
view.camera = scene.TurntableCamera(up='z', fov=60)

print(canvas.scene.describe_tree())

# Simple surface plot example
# x, y values are not specified, so assumed to be 0:50
z = np.random.normal(size=(250, 250), scale=200)
z[100, 100] += 50000
z = gaussian_filter(z, (10, 10))
p1 = scene.visuals.SurfacePlot(z=z, color=(0.3, 0.3, 1, 1))
p1.transform = scene.transforms.MatrixTransform()
p1.transform.scale([1/249., 1/249., 1/249.])
p1.transform.translate([-0.5, -0.5, 0])

view.add(p1)

# p1._update_data()  # cheating.
# cf = scene.filters.ZColormapFilter('fire', zrange=(z.max(), z.min()))
# p1.attach(cf)


xax = scene.Axis(pos=[[-0.5, -0.5], [0.5, -0.5]], tick_direction=(0, -1),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
xax.transform = scene.STTransform(translate=(0, 0, -0.2))

yax = scene.Axis(pos=[[-0.5, -0.5], [-0.5, 0.5]], tick_direction=(-1, 0),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
yax.transform = scene.STTransform(translate=(0, 0, -0.2))

# Add a 3D axis to keep us oriented
axis = scene.visuals.XYZAxis(parent=view.scene)

if __name__ == '__main__':
    canvas.show()
    if sys.flags.interactive == 0:
        app.run()

Make sure to use the add_widget and add_view methods when using Widget or a subclass of Widget so canvas events are handled for you properly. The neat thing about this approach is that you can have multiple Background instances rendered in a row, with each one being rendered on top of the previous, providing you with a multi-layered background where each background can be an arbitrary 2D or 3D scene.

Solution 2: Use translate and scale transforms on the image outside of a view

An alternative to clearing the OpenGL depth buffer would be to translate the Image so its z coordinate is as close to 1.0 as possible, but not actually equal to 1.0. This approach won't work with PanZoomCamera, though, so you will also have to handle the scaling of the image so its fits the entire canvas once the canvas is resized. The resize event is also called during initial canvas setup. Full code below:

import sys
import numpy as np

from vispy import app, scene
from vispy.util.filter import gaussian_filter
from vispy.visuals.transforms import STTransform

canvas = scene.SceneCanvas(keys='interactive', bgcolor='w')

background_image = scene.Image(np.random.rand(512, 512, 3).astype(np.float32), parent=canvas.central_widget)
canvas.events.resize.connect(lambda _: background_image.__setattr__('transform', STTransform(
    translate=(0, 0, 0.999999970197677556793536268742172978818416595458984374),
    scale=tuple(c / b for c, b in zip(canvas.size, background_image.size))
)))

view = canvas.central_widget.add_view()
view.camera = scene.TurntableCamera(up='z', fov=60)

# Simple surface plot example
# x, y values are not specified, so assumed to be 0:50
z = np.random.normal(size=(250, 250), scale=200)
z[100, 100] += 50000
z = gaussian_filter(z, (10, 10))
p1 = scene.visuals.SurfacePlot(z=z, color=(0.3, 0.3, 1, 1))
p1.transform = scene.transforms.MatrixTransform()
p1.transform.scale([1/249., 1/249., 1/249.])
p1.transform.translate([-0.5, -0.5, 0])

view.add(p1)

# p1._update_data()  # cheating.
# cf = scene.filters.ZColormapFilter('fire', zrange=(z.max(), z.min()))
# p1.attach(cf)


xax = scene.Axis(pos=[[-0.5, -0.5], [0.5, -0.5]], tick_direction=(0, -1),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
xax.transform = scene.STTransform(translate=(0, 0, -0.2))

yax = scene.Axis(pos=[[-0.5, -0.5], [-0.5, 0.5]], tick_direction=(-1, 0),
                 font_size=16, axis_color='k', tick_color='k', text_color='k',
                 parent=view.scene)
yax.transform = scene.STTransform(translate=(0, 0, -0.2))

# Add a 3D axis to keep us oriented
axis = scene.visuals.XYZAxis(parent=view.scene)

if __name__ == '__main__':
    canvas.show()
    if sys.flags.interactive == 0:
        app.run()

With this approach you don't have to worry about drawing the Image before the 3D scene. Any drawing order works. If the image doesn't show, its z coordinate is probably being rounded up to 1.0, so try to remove a few decimals in the translate parameter of STTransform. Values of 0.9999 and upwards should work well enough. I just went and found the maximum possible value I could put in the z coordinate that would display the image for me.