How to capture frames from Apple iSight using Python and PyObjC?

Question

I am trying to capture a single frame from the Apple iSight camera built into a Macbook Pro using Python (version 2.7 or 2.6) and the PyObjC (version 2.2).

As a starting point, I used this old StackOverflow question. To verify that it makes sense, I cross-referenced against Apple's MyRecorder example that it seems to be based on. Unfortunately, my script does not work.

My big questions are:

• Am I initializing the camera correctly?
• Am I starting the event loop correctly?
• Was there any other setup I was supposed to do?

In the example script pasted below, the intended operation is that after calling startImageCapture(), I should start printing "Got a frame..." messages from the CaptureDelegate. However, the camera's light never turns on and the delegate's callback is never executed.

Also, there are no failures during startImageCapture(), all functions claim to succeed, and it successfully finds the iSight device. Analyzing the session object in pdb shows that it has valid input and output objects, the output has a delegate assigned, the device is not in use by another processes, and the session is marked as running after startRunning() is called.

Here's the code:

#!/usr/bin/env python2.7

import sys
import os
import time
import objc
import QTKit
import AppKit
from Foundation import NSObject
from Foundation import NSTimer
from PyObjCTools import AppHelper
objc.setVerbose(True)

class CaptureDelegate(NSObject):
    def captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_(self, captureOutput, 
                                                                           videoFrame, sampleBuffer, 
                                                                           connection):
        # This should get called for every captured frame
        print "Got a frame: %s" % videoFrame

class QuitClass(NSObject):
    def quitMainLoop_(self, aTimer):
        # Just stop the main loop.
        print "Quitting main loop."
        AppHelper.stopEventLoop()


def startImageCapture():
    error = None

    # Create a QT Capture session
    session = QTKit.QTCaptureSession.alloc().init()

    # Find iSight device and open it
    dev = QTKit.QTCaptureDevice.defaultInputDeviceWithMediaType_(QTKit.QTMediaTypeVideo)
    print "Device: %s" % dev
    if not dev.open_(error):
        print "Couldn't open capture device."
        return

    # Create an input instance with the device we found and add to session
    input = QTKit.QTCaptureDeviceInput.alloc().initWithDevice_(dev)
    if not session.addInput_error_(input, error):
        print "Couldn't add input device."
        return

    # Create an output instance with a delegate for callbacks and add to session
    output = QTKit.QTCaptureDecompressedVideoOutput.alloc().init()
    delegate = CaptureDelegate.alloc().init()
    output.setDelegate_(delegate)
    if not session.addOutput_error_(output, error):
        print "Failed to add output delegate."
        return

    # Start the capture
    print "Initiating capture..."
    session.startRunning()


def main():
    # Open camera and start capturing frames
    startImageCapture()

    # Setup a timer to quit in 10 seconds (hack for now)
    quitInst = QuitClass.alloc().init()
    NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(10.0, 
                                                                             quitInst, 
                                                                             'quitMainLoop:', 
                                                                             None, 
                                                                             False)
    # Start Cocoa's main event loop
    AppHelper.runConsoleEventLoop(installInterrupt=True)

    print "After event loop"


if __name__ == "__main__":
    main()

Thanks for any help you can provide!

Answer 1

OK, I spent a day diving through the depths of PyObjC and got it working.

For future record, the reason the code in the question did not work: variable scope and garbage collection. The session variable was deleted when it fell out of scope, which happened before the event processor ran. Something must be done to retain it so it is not freed before it has time to run.

Moving everything into a class and making session a class variable made the callbacks start working. Additionally, the code below demonstrates getting the frame's pixel data into bitmap format and saving it via Cocoa calls, and also how to copy it back into Python's world-view as a buffer or string.

The script below will capture a single frame

#!/usr/bin/env python2.7
#
# camera.py -- by Trevor Bentley (02/04/2011)
# 
# This work is licensed under a Creative Commons Attribution 3.0 Unported License.
#
# Run from the command line on an Apple laptop running OS X 10.6, this script will
# take a single frame capture using the built-in iSight camera and save it to disk
# using three methods.
#

import sys
import os
import time
import objc
import QTKit
from AppKit import *
from Foundation import NSObject
from Foundation import NSTimer
from PyObjCTools import AppHelper

class NSImageTest(NSObject):
    def init(self):
        self = super(NSImageTest, self).init()
        if self is None:
            return None

        self.session = None
        self.running = True

        return self

    def captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_(self, captureOutput, 
                                                                           videoFrame, sampleBuffer, 
                                                                           connection):
        self.session.stopRunning() # I just want one frame

        # Get a bitmap representation of the frame using CoreImage and Cocoa calls
        ciimage = CIImage.imageWithCVImageBuffer_(videoFrame)
        rep = NSCIImageRep.imageRepWithCIImage_(ciimage)
        bitrep = NSBitmapImageRep.alloc().initWithCIImage_(ciimage)
        bitdata = bitrep.representationUsingType_properties_(NSBMPFileType, objc.NULL)

        # Save image to disk using Cocoa
        t0 = time.time()
        bitdata.writeToFile_atomically_("grab.bmp", False)
        t1 = time.time()
        print "Cocoa saved in %.5f seconds" % (t1-t0)

        # Save a read-only buffer of image to disk using Python
        t0 = time.time()
        bitbuf = bitdata.bytes()
        f = open("python.bmp", "w")
        f.write(bitbuf)
        f.close()
        t1 = time.time()
        print "Python saved buffer in %.5f seconds" % (t1-t0)

        # Save a string-copy of the buffer to disk using Python
        t0 = time.time()
        bitbufstr = str(bitbuf)
        f = open("python2.bmp", "w")
        f.write(bitbufstr)
        f.close()
        t1 = time.time()
        print "Python saved string in %.5f seconds" % (t1-t0)

        # Will exit on next execution of quitMainLoop_()
        self.running = False

    def quitMainLoop_(self, aTimer):
        # Stop the main loop after one frame is captured.  Call rapidly from timer.
        if not self.running:
            AppHelper.stopEventLoop()

    def startImageCapture(self, aTimer):
        error = None
        print "Finding camera"

        # Create a QT Capture session
        self.session = QTKit.QTCaptureSession.alloc().init()

        # Find iSight device and open it
        dev = QTKit.QTCaptureDevice.defaultInputDeviceWithMediaType_(QTKit.QTMediaTypeVideo)
        print "Device: %s" % dev
        if not dev.open_(error):
            print "Couldn't open capture device."
            return

        # Create an input instance with the device we found and add to session
        input = QTKit.QTCaptureDeviceInput.alloc().initWithDevice_(dev)
        if not self.session.addInput_error_(input, error):
            print "Couldn't add input device."
            return

        # Create an output instance with a delegate for callbacks and add to session
        output = QTKit.QTCaptureDecompressedVideoOutput.alloc().init()
        output.setDelegate_(self)
        if not self.session.addOutput_error_(output, error):
            print "Failed to add output delegate."
            return

        # Start the capture
        print "Initiating capture..."
        self.session.startRunning()


    def main(self):
        # Callback that quits after a frame is captured
        NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(0.1, 
                                                                                 self, 
                                                                                 'quitMainLoop:', 
                                                                                 None, 
                                                                                 True)

        # Turn on the camera and start the capture
        self.startImageCapture(None)

        # Start Cocoa's main event loop
        AppHelper.runConsoleEventLoop(installInterrupt=True)

        print "Frame capture completed."

if __name__ == "__main__":
    test = NSImageTest.alloc().init()
    test.main()

Answer 2

QTKit is deprecated and PyObjC is a big dependency (and seems to be tricky to build if you want it in HomeBrew). Plus PyObjC did not have most of AVFoundation so I created a simple camera extension for Python that uses AVFoundation to record a video or snap a picture, it requires no dependencies (Cython intermediate files are committed to avoid the need to have Cython for most users).

It should be possible to build it like this:

pip install -e git+https://github.com/dashesy/pyavfcam.git

Then we can use it to take a picture:

import pyavfcam

# Open the default video source
cam = pyavfcam.AVFCam(sinks='image')
frame = cam.snap_picture('test.jpg')  # frame is a memory buffer np.asarray(frame) can retrieve

Not related to this question, but if the AVFCam class is sub-classed, the overridden methods will be called with the result.