Efficient video rendering in a 3D plot using animation and blitting

Question

I'm currently doing a visualization task where I want to display a video within a 3D plot using matplotlib animation class. The purpose is to visualize feature tracking within the image.

I have a working video playing by repeatedly redrawing the figure but it works very slowly (around 1 fps) and I want to make it more efficient since I will add more drawings to the figure later.

I tried using blitting which is supposed to only draw data that has changed. I did it by settings blit=True within the animation class and reproducing what's done in the official documentation BUT the video does not appear to be playing at all. The only time it displays any images at all is when I resize the window which appears to force the window to update.

Why isn't it working? Am I going for the right approach to do it? Is there some other library than matplotlib that is more appropriate for this?

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from mpl_toolkits.mplot3d import Axes3D
import cv2 as cv


frames = []
for i in range(100):
    image = np.random.randint(0,255, size=(260,346), dtype="uint8")
    frames.append(image)

W, H = 260,346
X = np.arange(0, W)
Y = np.arange(0, H)
X, Y = np.meshgrid(X, Y)

fig = plt.figure(figsize=(10, 10))
ax = fig.add_subplot(111, projection='3d')
surf = ax.plot_surface(0, X, Y, cstride=1, rstride=1, 
                        linewidth=1, shade=False, edgecolor='none')

clen = (W - 1) * (H - 1)

def init():
    # formatting
    ax.get_proj = lambda: np.dot(Axes3D.get_proj(ax), 
                                    np.diag([1.2, 0.8, 0.8 *
float(H) / float(W), 1]))
    ax.view_init(elev=20)
    ax.invert_zaxis()

    ax.get_xaxis().set_visible(False)
    ax.xaxis.labelpad = 20
    ax.get_yaxis().set_visible(False)
    ax.zaxis.set_major_locator(plt.NullLocator())
    ax.yaxis.set_major_locator(plt.NullLocator())
    ax.set_xlabel("Time [s]", fontsize=15)
    ax.set_xlim([0, 0.5])
    ax.set_ylim([0, W - 1])
    ax.set_zlim([H - 1, 0])
    return surf,

def update(image):
    print("Updated...")
    img = cv.cvtColor(image, cv.COLOR_GRAY2RGB) / 255
    surf.set_facecolor(img[:-1, :-1].reshape(clen, 3))
    
    return surf,


#Create the Animation object
ani = animation.FuncAnimation(fig, update, frames, init_func = init,
                                interval=500, blit=True)
plt.show()

Answer 1

Okay so I never solved this but as an alternative solution I ran ani.save("test.gif", fps=23) and the resulting gif was shown as I expected it. I guess matplotlibs are not suitable at updating large amount of data at a low frame rate.