dex2oat Failing on Android 6/Marshmallow - Permission denied - Not instant run related

Question

In our production build (done on a CI server, so no Instant Run), users are reporting that the app either:

• takes a long time (> 10 minutes) to install, possibly never completing

or

• after rebooting, they see "Optimizing Apps 1 of 1" with our application installed.

With several bug reports from them at this point, the common thread seems to be the dex2oat process failing:

dex2oat : /system/bin/dex2oat --compiler-filter=speed --dex-file=/data/app/**packagename**-1/base.apk --oat-file=/data/dalvik-cache/arm64/data@app@**packagename**[email protected]@classes.dex E dex2oat : Failed to create oat file: /data/dalvik-cache/arm64/data@app@**packagename**[email protected]@classes.dex: Permission denied 05-11 20:48:21.783 7316 7316 I dex2oat : dex2oat took 3.058ms (threads: 8) 05-11 20:48:21.793 7293 7293 W art : Failed execv(/system/bin/dex2oat --runtime-arg -classpath --runtime-arg --instruction-set=arm64 --instruction-set-features=smp,a53 --runtime-arg -Xrelocate --boot-image=/system/framework/boot.art --runtime-arg -Xms64m --runtime-arg -Xmx512m --compiler-filter=speed --instruction-set-variant=generic --instruction-set-features=default --dex-file=/data/app/**packagename**-1/base.apk --oat-file=/data/dalvik-cache/arm64/data@app@**packagename**[email protected]@classes.dex) because non-0 exit status

We are seeing this issue across multiple manufacturers (Samsung, Kyocera, Blackberry) and phones. So far, all the bug reports that I have are against Android 6.0.X

• Gradle version: gradle-4.4
• Gradle plugin: 'com.android.tools.build:gradle:3.1.2'
• compileSdkVersion 26
• buildToolsVersion '27.0.3'
• minSdkVersion 19
• targetSdkVersion 26
• Multidex version 1.0.3

I'm aware of Several Bugs with similar symptoms around Instant Run, but I do not believe this is that. I've also seen a few other, older questions here, but only the Instant Run related ones have any answers or they are with much older versions of the plugins/gradle/build tools.

Comparing the last version we had before we started getting these reports to now, the only thing I think might be relevant is that I upgraded the Multidex library from 1.0.2 to 1.0.3.

Of course, I have been totally unable to reproduce the issue on any of my test devices, across any OS version, including factory resetting them.

Anyone have any thoughts, suggestions or even things to try? I'm running out of ideas quickly.

Additional information

Having finally gotten ahold of one of the devices where this problem is evident, some additional facts:

• The problem occurs installing via adb as well
• The problem occurs with a locally compiled binary
• Setting the binary to debuggable true causes the installation to go from 9:43 to less than 20 seconds. This is true even if I leave other settings such as proguard and minification turned on.

Answer 1

Looks like the Gradle Plugin 3.1.2 is causing the Issue

• https://issuetracker.google.com/issues/79731127
• https://issuetracker.google.com/issues/79962500

I was able to confirm that this is a bug in the android gradle plugin version 3.1.2. By reverting to 3.0.1 I was able to compile identical code and have it dex2oat immediately and successfully. This issue is also present in the 3.2.0-alpha15 gradle plugin.