Kodein + Ktor = mutation attempt of frozen kotlin.collections.HashMap - why?

Question

I have been struggling with this exception for couple of days recently.

I have a kotlin multiplatform project with these dependencies:

• kotlin=1.5.10
• kodein=7.6.0
• ktor=1.6.0 (uses kotlin coroutines 1.5.0-native-mt internally)

And I have been getting mentioned exception while trying to use httpClient in native:

    at kotlin.Throwable#<init>(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/Throwable.kt:23)
    at kotlin.Exception#<init>(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/Exceptions.kt:23)
    at kotlin.RuntimeException#<init>(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/Exceptions.kt:34)
    at kotlin.native.concurrent.InvalidMutabilityException#<init>(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:22)
    at <global>.ThrowInvalidMutabilityException(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:93)
    at <global>.MutationCheck(Unknown Source)
    at kotlin.collections.HashMap.<set-length>#internal(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/collections/HashMap.kt:16)
    at kotlin.collections.HashMap#addKey(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/collections/HashMap.kt:292)
    at kotlin.collections.HashMap#put(/Users/teamcity3/buildAgent/work/290aee0e088a1666/runtime/src/main/kotlin/kotlin/collections/HashMap.kt:68)
    at org.kodein.di.internal.DITreeImpl#find(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/internal/DITreeImpl.kt:132)
    at org.kodein.di.DITree#find$default(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/DITree.kt:36)
    at org.kodein.di.internal.DIContainerImpl#factory(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/internal/DIContainerImpl.kt:158)
    at org.kodein.di.DIContainer#factory$default(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/DIContainer.kt:32)
    at org.kodein.di.DIContainer#provider(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/DIContainer.kt:76)
    at org.kodein.di.DIContainer#provider$default(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/DIContainer.kt:75)
    at org.kodein.di.internal.DirectDIBaseImpl#Instance(/Users/runner/work/Kodein-DI/Kodein-DI/kodein-di/src/commonMain/kotlin/org/kodein/di/internal/DirectDIImpl.kt:30)
    at InvalidMutabilitySampleTest.<init>$lambda-6$lambda-5$lambda-4$lambda-3$lambda-2#internal(/Users/r.juszczyk/StudioProjects/lmhu-multiplatform-app/MultiplatformApp/src/iosTest/kotlin/InvalidMutabilitySampleTest.kt:26)
    at InvalidMutabilitySampleTest.$<init>$lambda-6$lambda-5$lambda-4$lambda-3$lambda-2$FUNCTION_REFERENCE$3.invoke#internal(/Users/r.juszczyk/StudioProjects/lmhu-multiplatform-app/MultiplatformApp/src/iosTest/kotlin/InvalidMutabilitySampleTest.kt:25)
    at InvalidMutabilitySampleTest.$<init>$lambda-6$lambda-5$lambda-4$lambda-3$lambda-2$FUNCTION_REFERENCE$3.$<bridge-UNNN>invoke(/Users/r.juszczyk/StudioProjects/lmhu-multiplatform-app/MultiplatformApp/src/iosTest/kotlin/InvalidMutabilitySampleTest.kt:25)
    at io.ktor.client.HttpClientConfig.install$<anonymous>_1#internal(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:69)
    at io.ktor.client.HttpClientConfig.$install$<anonymous>_1$FUNCTION_REFERENCE$17.invoke#internal(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:65)
    at io.ktor.client.HttpClientConfig.$install$<anonymous>_1$FUNCTION_REFERENCE$17.$<bridge-UNNN>invoke(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:65)
    at io.ktor.client.features.json.JsonFeature.Feature#prepare(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-features/ktor-client-json/common/src/io/ktor/client/features/json/JsonFeature.kt:129)
    at io.ktor.client.HttpClientConfig.install$<anonymous>_1-2#internal(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:77)
    at io.ktor.client.HttpClientConfig.$install$<anonymous>_1-2$FUNCTION_REFERENCE$18.invoke#internal(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:74)
    at io.ktor.client.HttpClientConfig.$install$<anonymous>_1-2$FUNCTION_REFERENCE$18.$<bridge-UNNN>invoke(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:74)
    at io.ktor.client.HttpClientConfig#install(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClientConfig.kt:97)
    at io.ktor.client.HttpClient#<init>(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClient.kt:172)
    at io.ktor.client.HttpClient#<init>(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClient.kt:81)
    at io.ktor.client#HttpClient(/Users/administrator/Documents/agent/work/8d547b974a7be21f/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClient.kt:43)

I managed to reproduce that crash in a test:

class InvalidMutabilitySampleTest {
    val di = DI {
        import(DI.Module("Some Module") {
            bind<Json>() with provider {
                Json {
                    prettyPrint = true
                    isLenient = true
                }
            }

            bind<HttpClient>() with provider {
                HttpClient{
                    install(JsonFeature) {
                        serializer = KotlinxSerializer(instance())
                    }
                }
            }
        })
    }

    val httpClient: HttpClient by di.instance()

    @Test
    fun invalidMutabilityTest() {
        println(httpClient)
    }

}

I also managed to fix it by changing:

HttpClient{
   install(JsonFeature) {
      serializer = KotlinxSerializer(instance())
   }
}

to:

val json = instance<Json>()
HttpClient{
   install(JsonFeature) {
      serializer = KotlinxSerializer(json)
   }
}

Then I noticed that HttpClient does a very specific thing - it freezes itself in the init block. I managed to reproduce it with this sample code:

class FrozenConstructor(val block: ()->Unit) {
    init {
        freeze()
    }
}

class InvalidMutabilitySampleTest2 {
    val di = DI {
        import(DI.Module("Some Module") {
            bind<String>() with provider {
                "lolo"
            }

            bind<FrozenConstructor>() with provider {
                FrozenConstructor{
                    instance<String>()
                }
            }
        })
    }

    val frozenConstructor: FrozenConstructor by di.instance()

    @Test
    fun invalidMutabilityTest() {
        println(frozenConstructor)
    }
}

So my theory is following:

1. kodein tries to provide FrozenConstructor class
2. FrozenConstructor is created and it freezes itself, and its member which references kodein
3. kodein tries to cache provided dependency and try to mutate internal MutableMap which is frozen and everything crashes

Can someone confirm that this is more or less correct , and correct me if not?

And also can you guys suggest the best way of handling it, and other pitfalls which are waiting there?

Is it a kodein bug?

Why kodein has to store something in a mutable map if I am using with provider not with singleton ?

Answer 1

I think you've pretty much figured it out. Ktor freezes itself and all of its configuration in order to ensure that it can be used across threads in Kotlin/Native. Kodein assumes you will only ever touch it from one thread and is not safe to freeze. (Whether that's a limitation, a bug, or a design flaw is maybe up for interpretation.)

To work around these issues, you want to avoid accidentally capturing this references in your HttpClient configuration that refer to Kodein internals. A good way to do this is to grab instances from DI in helper variables outside the HttpClient lambda, as you've discovered.

Answer 2

You are correct. FrozenConstructor freezes itself, and therefore its block property, which, as a lambda, freezes all its captures, which includes the DI container.

Kodein-DI does not support freezing (and therefore native multi-threading). The decision has been made not to invest in making it compatible with native multi-thrading since JB has announced working on a new GC.