Capture Webcam image using CV2 and Pyglet in Python

Question

I'm using CV2 (OpenCV) for Python, and the Pyglet Python libraries to create a small application which will display live video from a webcam and have some text or static images overlayed. I've already made an application with CV2 that just displays the webcam image in a frame, but now I'd like to get that frame inside a pyglet window.

Here's what I've cobbled together so far:

import pyglet
from pyglet.window import key
import cv2
import numpy


window = pyglet.window.Window()

camera=cv2.VideoCapture(0)

def getCamFrame(color,camera):
    retval,frame=camera.read()
    if not color:
        frame=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    frame=numpy.rot90(frame)
    return frame


frame=getCamFrame(True,camera)
video = pyglet.resource.media(frame, streaming=True)

@window.event
def on_key_press(symbol, modifiers):
    if symbol == key.ESCAPE:
        print 'Application Exited with Key Press'
        window.close()

@window.event
def on_draw():
    window.clear()
    video.blit(10,10)

pyglet.app.run()

When run, I get the following error:

Traceback, line 20 in <module>
  video = pyglet.resource.media(frame, streaming=True)
TypeError: unhashable type: 'numpy.ndarray'

I'm also open to other options that would let me display text over my live video. I originally used pygame, but in the end, I'll need multiple monitor support, so that's why I'm using pyglet.

Answer 1

There are a number of problems with your approach, but the trickiest thing is converting numpy arrays to textures. I use the approach below, which I discovered at some point elsewhere on SO. In short, you have to utilize the ctypes types and structures exposed by pyglet.gl in order to generate an array of GLubytes, and then put the contents of the image (a numpy array) into it. Then, because you have a 1-d array of values, you have to specify how Pyglet should make the image, pImage here, by specifying the pixel format and pitch.

If you get the example below working, you should be able to get pImg to update on each call of on_draw, and you should be done.

import pyglet
from pyglet.gl import *
from pyglet.window import key
import cv2
import numpy
import sys

window = pyglet.window.Window()

camera=cv2.VideoCapture(0)

retval,img = camera.read()
sy,sx,number_of_channels = img.shape
number_of_bytes = sy*sx*number_of_channels

img = img.ravel()

image_texture = (GLubyte * number_of_bytes)( *img.astype('uint8') )
# my webcam happens to produce BGR; you may need 'RGB', 'RGBA', etc. instead
pImg = pyglet.image.ImageData(sx,sy,'BGR',
       image_texture,pitch=sx*number_of_channels)

@window.event
def on_key_press(symbol, modifiers):
    if symbol == key.ESCAPE:
        print 'Application Exited with Key Press'
        window.close()

@window.event
def on_draw():
    window.clear()
    pImg.blit(0,0)

pyglet.app.run()

Answer 2

import pyglet
import cv2
window = pyglet.window.Window()

video = cv2.VideoCapture(0)
def takepicture(dt):
    num = 0

    ret,frame = video.read()
    cv2.imwrite(str(num)+'.jpg',frame)
    print("Image_Captured")

@window.event
def on_draw():
    window.clear()
    image = pyglet.image.load('0.jpg')
    image.blit(0,0)

pyglet.clock.schedule_interval(takepicture, 0.001)


pyglet.app.run()

Answer 3

You can convert each opencv image to a pyglet image by using the ImageData constructor. The idea is to convert the opencv image to a PIL array, which in turn is converted to a string of bytes, and then passed as the raw data to the constructor.

from PIL import Image
def cv2glet(img):
    '''Assumes image is in BGR color space. Returns a pyimg object'''
    rows, cols, channels = img.shape
    raw_img = Image.fromarray(img).tobytes()

    top_to_bottom_flag = -1
    bytes_per_row = channels*cols
    pyimg = pyglet.image.ImageData(width=cols, 
                                   height=rows, 
                                   format='BGR', 
                                   data=raw_img, 
                                   pitch=top_to_bottom_flag*bytes_per_row)
    return pyimg

Answer 4

While this works, I found that loading images from numpy arrays was pretty slow when the image was high resolution. pygarrrayimage, a python module on github, can load numpy arrays directly into the video card without making a copy:

https://github.com/motmot/pygarrayimage

This kept my python application which is loading images from high res videos from lagging up. Check out the example folder for how to blit images to the screen fast.