YUV (NV21) to BGR conversion on mobile devices (Native Code)

Question

I'm developing a mobile application that runs on Android and IOS. It's capable of real-time-processing of a video stream. On Android I get the Preview-Videostream of the camera via android.hardware.Camera.PreviewCallback.onPreviewFrame. I decided to use the NV21-Format, since it should be supported by all Android-devices, whereas RGB isn't (or just RGB565).

For my algorithms, which mostly are for pattern recognition, I need grayscale images as well as color information. Grayscale is not a problem, but the color conversion from NV21 to BGR takes way too long.

As described, I use the following method to capture the images;

In the App, I override the onPreviewFrame-Handler of the Camera. This is done in CameraPreviewFrameHandler.java:

 @Override
      public void onPreviewFrame(byte[] data, Camera camera) {
      {
          try {
              AvCore.getInstance().onFrame(data, _prevWidth, _prevHeight, AvStreamEncoding.NV21);
          } catch (NativeException e) 
          {
              e.printStackTrace();
          } 
      }

The onFrame-Function then calls a native function which fetches data from the Java-Objects as local references. This is then converted to an unsigned char* bytestream and calls the following c++ function, which uses OpenCV to convert from NV21 to BGR:

void CoreManager::api_onFrame(unsigned char* rImageData, avStreamEncoding_t vImageFormat, int vWidth, int vHeight)
    {
    // rImageData is a local JNI-reference to the java-byte-array "data" from onPreviewFrame
    Mat bgrMat; // Holds the converted image
    Mat origImg;    // Holds the original image (OpenCV-Wrapping around rImageData)

    double ts; // for profiling
    switch(vImageFormat)
    {
           // other formats
        case NV21:
            origImg = Mat(vHeight + vHeight/2, vWidth, CV_8UC1, rImageData); // fast, only creates header around rImageData
            bgrMat = Mat(vHeight, vWidth, CV_8UC3); // Prepare Mat for target image
            ts = avUtils::gettime();                // PROFILING START
            cvtColor(origImg, bgrMat, CV_YUV2BGRA_NV21);
            _onFrameBGRConversion.push_back(avUtils::gettime()-ts); // PROFILING END
            break;
    }

    [...APPLICATION LOGIC...]
}

As one might conclude from comments in the code, I profiled the conversion already and it turned out that it takes ~30ms on my Nexus 4, which is unacceptable long for such a "trivial" pre-processing step. (My profiling methods are double-checked and working properly for real-time measurement)

Now I'm trying desperately to find a faster implementation of this color conversion from NV21 to BGR. This is what I've already done;

1. Adopted the code "convertYUV420_NV21toRGB8888" to C++ provided in this topic (multiple of the conversion-time)
2. Modified the code from 1 to use only integer operations (doubled conversion-time of openCV-Solution)
3. Browsed through a couple other implementations, all with similar conversion-times
4. Checked OpenCV-Implementation, they use a lot of bit-shifting to get performance. Guess I'm not able to do better on my own

Do you have suggestions / know good implementations or even have a completely different way to work around this Problem? I somehow need to capture RGB/BGR-Frames from the Android-Camera and it should work on as many Android-devices as possible.

Thanks for your replies!

Answer 1

Did you try libyuv? I used it in the past and if you compile it with NEON support, it uses an asm code optimized for ARM processors, you can start from there to further optimize for your special situation.