ML Kit Barcode process fails due to Image is already closed

Question

I'm looking for a few days how I can read a QR code using ML Kit and CameraX. The process of the input image returns always a failure "IllegalStateException: Image is already closed". I have looked on different forums, but I can not find what I'm doing wrong. I think I'm following the Google documentation, but it won't work.

This is what I'm doing in my MainActivity:

1. I create a textureView which shows me the input of my camera
2. I start my camera (after checking the necessary rights)
3. I setup a "preview" use case
4. I setup a "Image Analysis" use case
5. I set an analyzer which creates an InputImage which is passed to the Barcode scanner process
6. Set my LifeCycle with the use cases "preview" and "image analysis"

I'm using CameraX version "1.0.0-alpha04" and "com.google.mlkit:barcode-scanning:16.0.3".

build.gradle

apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.2"

    defaultConfig {
        applicationId "be.lapit.qrcodescanner"
        minSdkVersion 21
        targetSdkVersion 30
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    implementation 'androidx.core:core-ktx:1.3.2'
    implementation 'androidx.appcompat:appcompat:1.2.0'
    implementation 'androidx.constraintlayout:constraintlayout:2.0.1'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'

    // ML Kit Barcode scanner
    implementation 'com.google.mlkit:barcode-scanning:16.0.3'
    implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:16.1.2'

    // CameraX
    def camerax_version = "1.0.0-alpha04"
    // The following line is optional, as the core library is included indirectly by camera-camera2
    implementation "androidx.camera:camera-core:${camerax_version}"
    implementation "androidx.camera:camera-camera2:${camerax_version}"
    // If you want to additionally use the CameraX Lifecycle library
    //implementation "androidx.camera:camera-lifecycle:${camerax_version}"
    // If you want to additionally use the CameraX View class
    //implementation "androidx.camera:camera-view:1.0.0-alpha17"

}

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   xmlns:app="http://schemas.android.com/apk/res-auto"
   android:layout_width="match_parent"
   android:layout_height="match_parent">

   <LinearLayout
       android:id="@+id/linearLayout"
       android:layout_width="wrap_content"
       android:layout_height="wrap_content"
       android:gravity="center"
       android:orientation="vertical"
       app:layout_constraintBottom_toBottomOf="parent"
       app:layout_constraintEnd_toEndOf="parent"
       app:layout_constraintStart_toStartOf="parent">


       <ImageView
           android:id="@+id/imageView"
           android:layout_width="250px"
           android:layout_height="250px"
           android:layout_marginBottom="48dp"
           android:src="@drawable/thumb_up" />

       <TextView
           android:layout_width="wrap_content"
           android:layout_height="wrap_content"
           android:text="Test textview"/>

       <TextureView
           android:id="@+id/texture_view"
           android:layout_width="600px"
           android:layout_height="600px" />


   </LinearLayout>

</androidx.constraintlayout.widget.ConstraintLayout>

MainActivity.kt

package be.lapit.qrcodescanner

import android.Manifest
import android.content.Intent
import android.content.pm.PackageManager
import android.os.Bundle
import android.util.Log
import android.util.Size
import android.view.TextureView
import android.widget.Button
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import androidx.camera.core.*
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import androidx.lifecycle.LifecycleOwner
import com.google.android.gms.tasks.OnFailureListener
import com.google.mlkit.vision.barcode.Barcode
import com.google.mlkit.vision.barcode.BarcodeScannerOptions
import com.google.mlkit.vision.barcode.BarcodeScanning
import com.google.mlkit.vision.common.InputImage
import java.time.Instant
import java.time.format.DateTimeFormatter
import kotlin.system.measureTimeMillis


class MainActivity : AppCompatActivity() {

    // PLA: Variable needed to check if user has granted camera permission
    companion object {
        private const val REQUEST_CAMERA_PERMISSION = 10
    }

    private lateinit var textureView: TextureView

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        textureView = findViewById(R.id.texture_view)

        // Request camera permissions
        if (isCameraPermissionGranted()) {
            textureView.post { startCamera() }
        } else {
            ActivityCompat.requestPermissions(
                this,
                arrayOf(Manifest.permission.CAMERA),
                REQUEST_CAMERA_PERMISSION
            )
        }


    }

    private fun startCamera() {
        // -----------------------
        // Setup use case Preview
        // -----------------------
        val previewConfig : PreviewConfig = PreviewConfig.Builder()
            // We want to show input from front camera of the device
            .setLensFacing(CameraX.LensFacing.FRONT)
            .build()

        val preview = Preview(previewConfig)

        preview.setOnPreviewOutputUpdateListener { previewOutput ->
            textureView.setSurfaceTexture(previewOutput.surfaceTexture)
        }

        // ------------------------------
        // Setup use case Image Analysis
        // ------------------------------
        val imageAnalysisConfig = ImageAnalysisConfig.Builder()
            .setTargetResolution(Size(1280, 720))
            .setLensFacing(CameraX.LensFacing.FRONT)
            .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
            .setImageQueueDepth(1)
            .build()

        val imageAnalysis = ImageAnalysis(imageAnalysisConfig)

        // -------------------
        // Set image analyzer
        // -------------------
        imageAnalysis.setAnalyzer { imageProxy: ImageProxy, rotationDegrees: Int ->
            Log.i("Status", "In setAnalyzer on " + System.currentTimeMillis())

            val mediaImage = imageProxy.image
            if (mediaImage != null) {
                val image1 = InputImage.fromMediaImage(mediaImage, rotationDegrees)

                // --------------------------
                // Configure barcode scanner
                // --------------------------
                val barcodeScannerOptions: BarcodeScannerOptions = BarcodeScannerOptions.Builder()
                    .setBarcodeFormats(
                        Barcode.FORMAT_QR_CODE,
                        Barcode.FORMAT_AZTEC,
                        Barcode.FORMAT_EAN_13

                    )
                    .build()

                // Get an instance of BarcodeScanner
                val scanner = BarcodeScanning.getClient(barcodeScannerOptions)

                scanner.process(image1)
                    .addOnSuccessListener ({ barcodes ->
                        // Task completed successfully
                        Log.i("Status", "In success listener on timestamp " + System.currentTimeMillis())
                        for (barcode in barcodes) {
                            Log.i("Barcode", barcode.toString())
                        }
                        Toast.makeText(this,"Ik ben hier eindelijk",Toast.LENGTH_LONG).show();

                    })
                    .addOnFailureListener(OnFailureListener {
                        // Process failed
                        Log.i("Status", "In failure listener on " +  System.currentTimeMillis() + " because: $it")
                        Log.i("Failure", "$it")
                        it.printStackTrace()
                    })
                    .addOnCompleteListener {
                        Log.i("Status", "In complete listener on " +  System.currentTimeMillis())
                        //imageProxy.close()
                    }
            }
        }

        // Bind
        CameraX.bindToLifecycle(this as LifecycleOwner, preview, imageAnalysis)
    }


    private fun isCameraPermissionGranted(): Boolean {
        val selfPermission = ContextCompat.checkSelfPermission(
            baseContext,
            Manifest.permission.CAMERA
        )
        return selfPermission == PackageManager.PERMISSION_GRANTED
    }

    override fun onRequestPermissionsResult(
        requestCode: Int,
        permissions: Array<String>,
        grantResults: IntArray
    ) {
        if (requestCode == REQUEST_CAMERA_PERMISSION) {
            if (isCameraPermissionGranted()) {
                textureView.post { startCamera() }
            } else {
                Toast.makeText(this, "Camera permission is required.", Toast.LENGTH_SHORT).show()
                finish()
            }
        }
    }
}

Logging info

I/Status: In setAnalyzer on 1602190069609
I/tflite: Initialized TensorFlow Lite runtime.
I/native: barcode_detector_client.cc:238 Not using NNAPI
I/Status: In failure listener on 1602190069612 because: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
I/Failure: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
W/System.err: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
        at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@17.0.0:32)
        at com.google.mlkit.common.sdkinternal.zzl.run(com.google.mlkit:common@@17.0.0)
W/System.err:     at com.google.mlkit.common.sdkinternal.zzp.run(com.google.mlkit:common@@17.0.0:3)
        at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zzd(com.google.mlkit:common@@17.0.0:24)
        at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zza(com.google.mlkit:common@@17.0.0:30)
        at com.google.mlkit.common.sdkinternal.zzh.run(com.google.mlkit:common@@17.0.0)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:762)
    Caused by: java.lang.IllegalStateException: Image is already closed
        at android.media.Image.throwISEIfImageIsInvalid(Image.java:68)
        at android.media.ImageReader$SurfaceImage$SurfacePlane.getBuffer(ImageReader.java:803)
        at com.google.mlkit.vision.barcode.internal.zzf.zza(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:72)
        at com.google.mlkit.vision.barcode.internal.zzf.zzb(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:83)
        at com.google.mlkit.vision.barcode.internal.zzf.run(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:143)
        at com.google.mlkit.vision.common.internal.MobileVisionBase.zza(com.google.mlkit:vision-common@@16.1.0:23)
        at com.google.mlkit.vision.common.internal.zzc.call(com.google.mlkit:vision-common@@16.1.0)
        at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@17.0.0:29)
        ... 8 more
I/Status: In complete listener on 1602190069614
    In failure listener on 1602190069614 because: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
I/Failure: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
W/System.err: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
        at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@17.0.0:32)
        at com.google.mlkit.common.sdkinternal.zzl.run(com.google.mlkit:common@@17.0.0)
        at com.google.mlkit.common.sdkinternal.zzp.run(com.google.mlkit:common@@17.0.0:3)
        at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zzd(com.google.mlkit:common@@17.0.0:24)
        at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zza(com.google.mlkit:common@@17.0.0:30)
        at com.google.mlkit.common.sdkinternal.zzh.run(com.google.mlkit:common@@17.0.0)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:762)
W/System.err: Caused by: java.lang.IllegalStateException: Image is already closed
        at android.media.Image.throwISEIfImageIsInvalid(Image.java:68)
        at android.media.ImageReader$SurfaceImage$SurfacePlane.getBuffer(ImageReader.java:803)
        at com.google.mlkit.vision.barcode.internal.zzf.zza(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:72)
        at com.google.mlkit.vision.barcode.internal.zzf.zzb(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:83)
        at com.google.mlkit.vision.barcode.internal.zzf.run(com.google.android.gms:play-services-mlkit-barcode-scanning@@16.1.2:143)
        at com.google.mlkit.vision.common.internal.MobileVisionBase.zza(com.google.mlkit:vision-common@@16.1.0:23)
        at com.google.mlkit.vision.common.internal.zzc.call(com.google.mlkit:vision-common@@16.1.0)
        at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@17.0.0:29)
        ... 8 more
I/Status: In complete listener on 1602190069615

Answer 1

Just to help other people, I post my latest (working) code here

Build.gradle

apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.2"

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

    defaultConfig {
        applicationId "be.lapit.qrcodescanner"
        minSdkVersion 21
        targetSdkVersion 30
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    implementation 'androidx.core:core-ktx:1.3.2'
    implementation 'androidx.appcompat:appcompat:1.2.0'
    implementation 'androidx.constraintlayout:constraintlayout:2.0.1'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'

    // ML Kit Barcode scanner
    implementation 'com.google.mlkit:barcode-scanning:16.0.3'
    implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:16.1.2'

    // CameraX
    def camerax_version = "1.0.0-alpha10"
    // The following line is optional, as the core library is included indirectly by camera-camera2
    implementation "androidx.camera:camera-core:${camerax_version}"
    implementation "androidx.camera:camera-camera2:${camerax_version}"
    // If you want to additionally use the CameraX Lifecycle library
    implementation "androidx.camera:camera-lifecycle:${camerax_version}"
    // If you want to additionally use the CameraX View class
    implementation "androidx.camera:camera-view:1.0.0-alpha17"
    // If you want to additionally use the CameraX Extensions library
    implementation "androidx.camera:camera-extensions:1.0.0-alpha17"

}

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <LinearLayout
        android:id="@+id/linearLayout"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:gravity="center"
        android:orientation="vertical"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent">


        <Button
            android:id="@+id/btnGallery"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Get picture"></Button>

        <FrameLayout
            android:id="@+id/container"
            android:layout_width="450px"
            android:layout_height="450px">
            <androidx.camera.view.PreviewView
                android:id="@+id/preview_view"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:background="@color/colorRed"
                android:layout_gravity="center" />
        </FrameLayout>


    </LinearLayout>

</androidx.constraintlayout.widget.ConstraintLayout>

MainActivity.kt

package be.lapit.qrcodescanner

import android.Manifest
import android.content.Intent
import android.content.pm.PackageManager
import android.os.Bundle
import android.util.Log
import android.util.Size
import android.view.TextureView
import android.widget.Button
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import androidx.camera.core.*
import androidx.camera.core.impl.ImageAnalysisConfig
import androidx.camera.core.impl.PreviewConfig
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.view.PreviewView
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import androidx.lifecycle.LifecycleOwner
import com.google.android.gms.tasks.OnFailureListener
import com.google.common.util.concurrent.ListenableFuture
import com.google.mlkit.vision.barcode.Barcode
import com.google.mlkit.vision.barcode.BarcodeScannerOptions
import com.google.mlkit.vision.barcode.BarcodeScanning
import com.google.mlkit.vision.common.InputImage
import java.time.Instant
import java.time.format.DateTimeFormatter
import kotlin.system.measureTimeMillis


class MainActivity : AppCompatActivity() {

    // PLA: Variable needed to check if user has granted camera permission
    companion object {
        private const val REQUEST_CAMERA_PERMISSION = 10
    }

    private lateinit var cameraProviderFuture : ListenableFuture<ProcessCameraProvider>
    private lateinit var previewView: PreviewView

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        previewView = findViewById(R.id.preview_view)

        // Request camera permissions
        if (isCameraPermissionGranted()) {
            previewView.post { startCamera() }
        } else {
            ActivityCompat.requestPermissions(
                this,
                arrayOf(Manifest.permission.CAMERA),
                REQUEST_CAMERA_PERMISSION
            )
        }


    }

    private fun startCamera() {
        // -----------------------
        // Setup use case Preview
        // -----------------------
        cameraProviderFuture = ProcessCameraProvider.getInstance(this)

        cameraProviderFuture.addListener(Runnable {
            val cameraProvider = cameraProviderFuture.get()
            bindPreview(cameraProvider)
        }, ContextCompat.getMainExecutor(this))

    }


    fun bindPreview(cameraProvider : ProcessCameraProvider) {
        var preview : Preview = Preview.Builder()
            .build()

        var cameraSelector : CameraSelector = CameraSelector.Builder()
            .requireLensFacing(CameraSelector.LENS_FACING_FRONT)
            .build()

        var previewView: PreviewView = findViewById(R.id.preview_view)
        preview.setSurfaceProvider(previewView.surfaceProvider)

        val imageAnalysis = ImageAnalysis.Builder()
            .setTargetResolution(Size(1280, 720))
            .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
            .build()

        // Define options for barcode scanner
        val options = BarcodeScannerOptions.Builder()
            .setBarcodeFormats(
                Barcode.FORMAT_QR_CODE,
                Barcode.FORMAT_AZTEC)
            .build()

        imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), ImageAnalysis.Analyzer { imageProxy ->
            val rotationDegrees = imageProxy.imageInfo.rotationDegrees

            // Initiate barcode scanner
            val scanner = BarcodeScanning.getClient(options)

            // insert your code here.
            @androidx.camera.core.ExperimentalGetImage
            val mediaImage = imageProxy.image

            @androidx.camera.core.ExperimentalGetImage
            if (mediaImage != null) {
                val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)

                // Pass image to an ML Kit Vision API
                @androidx.camera.core.ExperimentalGetImage
                val result = scanner.process(image)
                    .addOnSuccessListener { barcodes ->
                        // Task completed successfully
                        Log.i("Status", "In success listener")
                        for (barcode in barcodes) {
                            val bounds = barcode.boundingBox
                            val corners = barcode.cornerPoints

                            val rawValue = barcode.rawValue
                            Log.i("QR code", rawValue.toString())

                            val valueType = barcode.valueType
                            // See API reference for complete list of supported types
                            when (valueType) {
                                Barcode.TYPE_WIFI -> {
                                    val ssid = barcode.wifi!!.ssid
                                    val password = barcode.wifi!!.password
                                    val type = barcode.wifi!!.encryptionType
                                }
                                Barcode.TYPE_URL -> {
                                    val title = barcode.url!!.title
                                    val url = barcode.url!!.url
                                }
                            }
                        }
                    }
                    .addOnFailureListener {
                        // Task failed with an exception
                        Log.i("Status", "In failure listener")
                    }
                    .addOnCompleteListener{
                        // Task failed with an exception
                        Log.i("Status", "In on complete listener")
                        imageProxy.close()
                    }
            }

        })

        cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)
    }


    private fun isCameraPermissionGranted(): Boolean {
        val selfPermission = ContextCompat.checkSelfPermission(
            baseContext,
            Manifest.permission.CAMERA
        )
        return selfPermission == PackageManager.PERMISSION_GRANTED
    }

    override fun onRequestPermissionsResult(
        requestCode: Int,
        permissions: Array<String>,
        grantResults: IntArray
    ) {
        if (requestCode == REQUEST_CAMERA_PERMISSION) {
            if (isCameraPermissionGranted()) {
                previewView.post { startCamera() }
            } else {
                Toast.makeText(this, "Camera permission is required.", Toast.LENGTH_SHORT).show()
                finish()
            }
        }
    }
}

Answer 2

I am not 100% sure whether this is related, but could you try updating your CameraX dependencies to the latest version? Yours is 1.0.0-alpha04, while the latest is 1.0.0-beta10. https://developer.android.com/jetpack/androidx/releases/camera

For the API you are using - setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE), I cannot find it any more on the CameraX dev site: https://developer.android.com/reference/androidx/camera/core/ImageAnalysis

As for .setImageQueueDepth(1), seems it doesn't work for the default strategy: https://developer.android.com/reference/androidx/camera/core/ImageAnalysis.Builder#setImageQueueDepth(int)

Another small suggestion: you currently create a new BarcodeDetector in each frame, which could be slow. Creating once outside the analyzer and reuse the same instance could be better.

Hope some of the points help :)

Answer 3

scanner.process(image) returns Task object, and you need to wait this task to end, before leaving. Otherwise image can be closed before it is analyzed:

Task<List<Barcode>> task = scanner.process(image)
Tasks.await(task);