How-to convert an iOS camera image to greyscale using the Accelerate Framework?

Question

It seems like this should be simpler than I'm finding it to be.

I have an AVFoundation frame coming back in the standard delegate method:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection

where I would like to convert the frame to greyscale using the Accelerate.Framework.

There is a family of conversion methods in the framework, including vImageConvert_RGBA8888toPlanar8(), which looks like it might be what I would like to see, however, I can't find any examples of how to use them!

So far, I have the code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{

      @autoreleasepool {
            CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
            /*Lock the image buffer*/
            CVPixelBufferLockBaseAddress(imageBuffer,0);
            /*Get information about the image*/
            uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
            size_t width = CVPixelBufferGetWidth(imageBuffer);
            size_t height = CVPixelBufferGetHeight(imageBuffer);
            size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);

            // vImage In
            Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
            const vImage_Buffer inImage = { bitmap, height, width, stride };

            //How can I take this inImage and convert it to greyscale?????
            //vImageConvert_RGBA8888toPlanar8()??? Is the correct starting format here??
      }    
}

So I have two questions: (1) In the code above, is RBGA8888 the correct starting format? (2) How can I actually make the Accelerate.Framework call to convert to greyscale?

Answer 1

There is an easier option here. If you change the camera acquire format to YUV, then you already have a greyscale frame that you can use as you like. When setting up your data output, use something like:

dataOutput.videoSettings = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) };

You can then access the Y plane in your capture callback using:

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
uint8_t *yPlane = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);

... do stuff with your greyscale camera image ...

CVPixelBufferUnlockBaseAddress(pixelBuffer);

Answer 2

The vImage method is to use vImageMatrixMultiply_Planar8 and a 1x3 matrix. vImageConvert_RGBA8888toPlanar8 is the function you use to convert a RGBA8888 buffer into 4 planar buffers. These are used by vImageMatrixMultiply_Planar8. vImageMatrixMultiply_ARGB8888 will do it too in one pass, but your gray channel will be interleaved with three other channels in the result. vImageConvert_RGBA8888toPlanar8 itself doesn't do any math. All it does is separate your interleaved image into separate image planes.

If you need to adjust the gamma as well, then probably vImageConvert_AnyToAny() is the easy choice. It will do the fully color managed conversion from your RGB format to a grayscale colorspace. See vImage_Utilities.h.

I like Tarks answer better though. It just leaves you in a position of having to color manage the Luminance manually (if you care).

Answer 3

(1) My experience with the iOS camera framework has been with images in the kCMPixelFormat_32BGRA format, which is compatible with the ARGB8888 family of functions. (It may be possible to use other formats as well.)

(2) The simplest way to convert from BGR to grayscale on iOS is to use vImageMatrixMultiply_ARGB8888ToPlanar8(): https://developer.apple.com/documentation/accelerate/1546979-vimagematrixmultiply_argb8888top

Here is a fairly complete example written in Swift. I'm assuming the Objective-C code would be similar.

        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
            // TODO: report error
            return
        }
        
        // Lock the image buffer
        if (kCVReturnSuccess != CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)) {
            // TODO: report error
            return
        }
        defer {
            CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
        }
        
        // Create input vImage_Buffer
        let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
        let width = CVPixelBufferGetWidth(imageBuffer)
        let height = CVPixelBufferGetHeight(imageBuffer)
        let stride = CVPixelBufferGetBytesPerRow(imageBuffer)
        var inImage = vImage_Buffer(data: baseAddress, height: UInt(height), width: UInt(width), rowBytes: stride)
        
        // Create output vImage_Buffer
        let bitmap = malloc(width * height)
        var outImage = vImage_Buffer(data: bitmap, height: UInt(height), width: UInt(width), rowBytes: width)
        defer {
            // Make sure to free unless the caller is responsible for this
            free(bitmap)
        }

        // Arbitrary divisor to scale coefficients to integer values
        let divisor: Int32 = 0x1000
        let fDivisor = Float(divisor)
        
        // Rec.709 coefficients
        var coefficientsMatrix = [
            Int16(0.0722 * fDivisor),  // blue
            Int16(0.7152 * fDivisor),  // green
            Int16(0.2126 * fDivisor),  // red
            0  // alpha
        ]

        // Convert to greyscale
        if (kvImageNoError != vImageMatrixMultiply_ARGB8888ToPlanar8(
            &inImage, &outImage, &coefficientsMatrix, divisor, nil, 0, vImage_Flags(kvImageNoFlags))) {
            // TODO: report error
            return
        }

The code above was inspired by a tutorial from Apple on grayscale conversion, which can be found at the following link. It also includes conversion to a CGImage if that is needed. Note that they assume RGB order instead of BGR, and they only provide a 3 coefficients instead of 4 (mistake?) https://developer.apple.com/documentation/accelerate/vimage/converting_color_images_to_grayscale

Answer 4

if you need to use BGRA vide streams - you can use this excellent conversion here

This is the function you'll need to take:

void neon_convert (uint8_t * __restrict dest, uint8_t * __restrict src, int numPixels)
      {
          int i;
          uint8x8_t rfac = vdup_n_u8 (77);
          uint8x8_t gfac = vdup_n_u8 (151);
          uint8x8_t bfac = vdup_n_u8 (28);
          int n = numPixels / 8;

          // Convert per eight pixels
          for (i=0; i < n; ++i)
          {
              uint16x8_t  temp;
              uint8x8x4_t rgb  = vld4_u8 (src);
              uint8x8_t result;

              temp = vmull_u8 (rgb.val[0],      bfac);
              temp = vmlal_u8 (temp,rgb.val[1], gfac);
              temp = vmlal_u8 (temp,rgb.val[2], rfac);

              result = vshrn_n_u16 (temp, 8);
              vst1_u8 (dest, result);
              src  += 8*4;
              dest += 8;
          }
      }

more optimisations (using assembly) are in the link

Answer 5

Convert BGRA Image to Grayscale with Accelerate vImage

This method is meant to illustrate getting Accelerate's vImage use in converting BGR images to grayscale. Your image may very well be in RGBA format and you'll need to adjust the matrix accordingly, but the camera outputs BGRA so I'm using it here. The values in the matrix are the same values used in OpenCV for cvtColor, there are other values you might play with like luminosity. I assume you malloc the appropriate amount of memory for the result. In the case of grayscale it is only 1-channel or 1/4 the memory used for BGRA. If anyone finds issues with this code please leave a comment.

Performance note

Converting to grayscale in this way may NOT be the fastest. You should check the performance of any method in your environment. Brad Larson's GPUImage might be faster, or even OpenCV's cvtColor. In any case you will want to remove the calls to malloc and free for the intermediate buffers and manage them for the app lifecycle. Otherwise, the function call will be dominated by the malloc and free. Apple's docs recommend reusing the whole vImage_Buffer when possible.

You can also read about solving the same problem with NEON intrinsics.

Finally, the fastest method is not converting at all. If you're getting image data from the device camera the device camera is natively in the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format. Meaning, grabbing the first plane's data (Y-Channel, luma) is the fastest way to get grayscale.

BGRA to Grayscale

- (void)convertBGRAFrame:(const CLPBasicVideoFrame &)bgraFrame toGrayscale:(CLPBasicVideoFrame &)grayscaleFrame
{
    vImage_Buffer bgraImageBuffer = {
        .width = bgraFrame.width,
        .height = bgraFrame.height,
        .rowBytes = bgraFrame.bytesPerRow,
        .data = bgraFrame.rawPixelData
    };

    void *intermediateBuffer = malloc(bgraFrame.totalBytes);
    vImage_Buffer intermediateImageBuffer = {
        .width = bgraFrame.width,
        .height = bgraFrame.height,
        .rowBytes = bgraFrame.bytesPerRow,
        .data = intermediateBuffer
    };

    int32_t divisor = 256;
//    int16_t a = (int16_t)roundf(1.0f * divisor);
    int16_t r = (int16_t)roundf(0.299f * divisor);
    int16_t g = (int16_t)roundf(0.587f * divisor);
    int16_t b = (int16_t)roundf(0.114f * divisor);
    const int16_t bgrToGray[4 * 4] = { b, 0, 0, 0,
                                       g, 0, 0, 0,
                                       r, 0, 0, 0,
                                       0, 0, 0, 0 };

    vImage_Error error;
    error = vImageMatrixMultiply_ARGB8888(&bgraImageBuffer, &intermediateImageBuffer, bgrToGray, divisor, NULL, NULL, kvImageNoFlags);
    if (error != kvImageNoError) {
        NSLog(@"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
    }

    vImage_Buffer grayscaleImageBuffer = {
        .width = grayscaleFrame.width,
        .height = grayscaleFrame.height,
        .rowBytes = grayscaleFrame.bytesPerRow,
        .data = grayscaleFrame.rawPixelData
    };

    void *scratchBuffer = malloc(grayscaleFrame.totalBytes);
    vImage_Buffer scratchImageBuffer = {
        .width = grayscaleFrame.width,
        .height = grayscaleFrame.height,
        .rowBytes = grayscaleFrame.bytesPerRow,
        .data = scratchBuffer
    };

    error = vImageConvert_ARGB8888toPlanar8(&intermediateImageBuffer, &grayscaleImageBuffer, &scratchImageBuffer, &scratchImageBuffer, &scratchImageBuffer, kvImageNoFlags);
    if (error != kvImageNoError) {
        NSLog(@"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
    }
    free(intermediateBuffer);
    free(scratchBuffer);
}

CLPBasicVideoFrame.h - For reference

typedef struct
{
    size_t width;
    size_t height;
    size_t bytesPerRow;
    size_t totalBytes;
    unsigned long pixelFormat;
    void *rawPixelData;
} CLPBasicVideoFrame;

I got through the grayscale conversion, but was having trouble with the quality when I found this book on the web called Instant OpenCV for iOS. I personally picked up a copy and it has a number of gems, although the code is bit of a mess. On the bright-side it is a very reasonably priced eBook.

I'm very curious about that matrix. I toyed around with it for hours trying to figure out what the arrangement should be. I would have thought the values should be on the diagonal, but the Instant OpenCV guys put it as above.