System error capturing the output of a MediaProjection virtual display to an ImageReader

Question

I am working on an application that needs to capture the screen to a bitmap to transmit. I am attempting to use the new Android 5.0 android.media.projection APIs to do the screen capture.

The workflow for this API culminates in a call to

mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, DPI,
   DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, surface, null, null);

In my initial attempt at this capture I sourced the surface object from a SurfaceView. This works correctly; the end result is a tiny duplicate of the display being drawn on-screen (resulting in a Droste Effect)

I thought the feature nearly complete, but I then discovered that SurfaceViews are (from a code standpoint) not readable; you cannot get a bitmap from them.

In looking for other solutions I came across this question which has a very similar goal to mine, and in that thread it is suggested to use an ImageReader instead of a SurfaceView to source the Surface that you pass to the createVirtualDisplay API call.

However, when I change my code to use an ImageReader in lieu of a SurfaceView I get runtime logcat errors (no exceptions), and the callback function for the ImageReader never gets called. The createVirtualDisplay call also returns a seemingly valid VirtualDisplay object.

Here is the logcat:

9230-9270/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: createGraphicBuffer failed
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count

That second line repeats ~100 times before it stops occurring.

Stepping through on the debugger I see that the first error occurs during the createVirtualDisplay call, and all the others happen some point after execution returns to system code.

The only meaningful result for this error relates to an issue in Kitkat, where the API I am trying to consume does not exist. Nonetheless, I tried the fix suggested here (putting android:hardwareAccelerated="false" in the manifest). This did not change the application's behavior.

How can I "set the buffer count" or otherwise work around this error and get the screen as a bitmap?

P.S. My development platform is the Nexus 6.

The full code block, as requested:

MediaProjection mediaProjection = mgr.getMediaProjection(resultCode, data);
ImageReader ir = ImageReader.newInstance(WIDTH, HEIGHT, ImageFormat.JPEG, 5);
VirtualDisplay v = mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, getApplicationContext().getResources().getDisplayMetrics().densityDpi, DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, ir.getSurface(), null, null);

Edit: Regarding the artifact issue, here is the code I am using to get the bitmap out of the image and display it:

 public void onImageAvailable(ImageReader reader) {
        Image image = null;
        ByteArrayOutputStream bos = null;

        try {
            image = reader.acquireLatestImage();
            if (null == image){
                return;
            }
            bos = new ByteArrayOutputStream();
            final Image.Plane[] planes = image.getPlanes();
            final ByteBuffer buffer = (ByteBuffer) planes[0].getBuffer().rewind();
            final Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(buffer);
            //bitmap.compress(Bitmap.CompressFormat.WEBP, 50, bos);

            runOnUiThread(new Runnable() {
                public void run() {
                    iv.setImageBitmap(bitmap);
                }
            });

Answer 1

I think I can answer this question now, I met the same problem and after I change ImageFormat.JPEG to PixelFormat.RGBA_8888 everything goes well. It seems ImageFormat.JPEG is not supported.

You need to use the following code to get the correct bitmap:

                    int width = img.getWidth();
                    int height = img.getHeight();
                    int pixelStride = planes[0].getPixelStride();
                    int rowStride = planes[0].getRowStride();
                    int rowPadding = rowStride - pixelStride * width;
                    byte[] newData = new byte[width * height * 4];

                    int offset = 0;
                    bitmap = Bitmap.createBitmap(metrics,width, height, Bitmap.Config.ARGB_8888);
                    ByteBuffer buffer = planes[0].getBuffer();
                    for (int i = 0; i < height; ++i) {
                        for (int j = 0; j < width; ++j) {
                            int pixel = 0;
                            pixel |= (buffer.get(offset) & 0xff) << 16;     // R
                            pixel |= (buffer.get(offset + 1) & 0xff) << 8;  // G
                            pixel |= (buffer.get(offset + 2) & 0xff);       // B
                            pixel |= (buffer.get(offset + 3) & 0xff) << 24; // A
                            bitmap.setPixel(j, i, pixel);
                            offset += pixelStride;
                        }
                        offset += rowPadding;
                    }

From this way, the content of bitmap is what you want.

PS: I really want to say, the doc of android is pretty bad. we need to investigate too much detail to use sdk api correctly.

Answer 2

A better way to get the Image from ImageReader is just to create right sized bitmap and use the method copyPixelsFromBuffer(). Create ImageReader as follows:

mImageReader = ImageReader.newInstance(mWidth, mHeight, ImageFormat.RGB_565, 2);

Then you can get the image from mImageReader using the code below.

final Image.Plane[] planes = image.getPlanes();
final ByteBuffer buffer = planes[0].getBuffer();
int offset = 0;
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * mWidth;
// create bitmap
bitmap = Bitmap.createBitmap(mWidth+rowPadding/pixelStride, mHeight, Bitmap.Config.RGB_565);
bitmap.copyPixelsFromBuffer(buffer);
image.close();

I have described the process of capturing screen using MediaProjection API along with the mistakes most people made when getting image from ImageReader in a blog post which you can read if interested.