Core Data migration process get stuck for a small number of users

Question

Abstract

I encountered a problem where Core Data migration process got stuck for a small number of users after updating the app.

Environment

I am using SwiftUI + Core Data, targeting iOS 15.0+.

Description

I'm not sure about the cause and couldn't reproduce it locally.

The corresponding modification to the model is adding a non-optional Int64 field to two entities, with no other changes. The crash log of the user corresponds to the line container.loadPersistentStores(completionHandler: { (storeDescription, error) in, so I assume the error occurred during migration.

What I have been tried

I have been tried to reproduce the crash in my local environment multiple times but failed. The migration process in my environment seems fine to me.

Possibilities that have been ruled out are:

1. Failure caused by simultaneous migration of the main app and the widget. I added a logic to delay widget migration if the migration is still in progress, but the crash still exist. And some users who don't use the widget still encounter this problem.
2. Insufficient storage space. Some users have more than 200G of free space but still encounter this issue.

I have set the default value for the new non-optional Int64, here is the related git diff for the model changes. The bolded 'order' fields are the new fields I added.

Related Code

    <entity name="Preset" representedClassName="Preset" syncable="YES">
        <attribute name="createTime" optional="YES" attributeType="Date" usesScalarValueType="NO"/>
        <attribute name="detail" optional="YES" attributeType="String"/>
        **<attribute name="order" attributeType="Integer 64" minValueString="0" defaultValueString="0" usesScalarValueType="YES"/>**
        <relationship name="belongsTo" optional="YES" maxCount="1" deletionRule="Nullify" destinationEntity="Project" inverseName="presets" inverseEntity="Project"/>
        <relationship name="tagBy" optional="YES" toMany="YES" deletionRule="Nullify" destinationEntity="Tag" inverseName="presets" inverseEntity="Tag"/>
    </entity>
...
        <attribute name="id_" optional="YES" attributeType="String"/>
        <attribute name="isArchived" optional="YES" attributeType="Boolean" usesScalarValueType="YES"/>
        <attribute name="name" optional="YES" attributeType="String"/>
        **<attribute name="order" attributeType="Integer 64" minValueString="0" defaultValueString="0" usesScalarValueType="YES"/>**
        <relationship name="entries" optional="YES" toMany="YES" deletionRule="Nullify" destinationEntity="TimeEntry" inverseName="belongsTo" inverseEntity="TimeEntry"/>
        <relationship name="presets" optional="YES" toMany="YES" deletionRule="Nullify" destinationEntity="Preset" inverseName="belongsTo" inverseEntity="Preset"/>
...

here is my Persistence.swift container initialization code:

init(inMemory: Bool = false) {
    container = NSPersistentCloudKitContainer(name: "Project")
    guard let groupContainerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.Project") else {
        logger.critical("Persistant init failed to get groupContainerURL")
        preconditionFailure("Persistant init failed to get groupContainerURL")
    }
    let storeURL = groupContainerURL.appendingPathComponent("Project.sqlite")
    if inMemory {
        container.persistentStoreDescriptions.first!.url = URL(fileURLWithPath: "/dev/null")
    } else {
        container.persistentStoreDescriptions = [NSPersistentStoreDescription(url: storeURL)]
    }
    
    if container.isNeedMigration(url: storeURL) {
        logger.notice("Core Data model need Migration")
        #if APP_WIDGET
        for idx in 0..<3 {
            logger.notice("Widget sleeping \(idx+1).")
            sleep(3)
            if !container.isNeedMigration(url: storeURL) {
                break
            }
        }
        #endif
    }
    container.loadPersistentStores(completionHandler: { (storeDescription, error) in
        if let error = error as NSError? {
            // Replace this implementation with code to handle the error appropriately.
            // fatalError() causes the application to generate a crash log and terminate. You should not use this function in a shipping application, although it may be useful during development.
            
            /*
             Typical reasons for an error here include:
             * The parent directory does not exist, cannot be created, or disallows writing.
             * The persistent store is not accessible, due to permissions or data protection when the device is locked.
             * The device is out of space.
             * The store could not be migrated to the current model version.
             Check the error message to determine what the actual problem was.
             */
            logger.critical("Unresolved error \(error.localizedDescription, privacy: .public), \(error.userInfo, privacy: .public), error: \(error)")
            fatalError("Unresolved error \(error.localizedDescription), \(error.userInfo)")
        }
    })
    container.viewContext.automaticallyMergesChangesFromParent = true
}

So I want to identify the reason of the crash and fix it, please give me some advice, thanks.

Crash Info

---

Update 1

This issue seems to be nasty now.I have been helped by two users, used some hacky way to get the logs while Persistence initializes (use HTTP POST method to directly post logs to my server), and found out that the crash is not initiated by the fatalError, but the internal of loadPersistentStores execution, the situation's details are described below:

1. I wrote a block HTTP POST request function to post logs to my server, which use semaphore to wait for the POST to finish.
2. I send logs from following positions:
   1. Beginning of Persistence init function
   2. Inside the isNeedMigration closure
   3. Before sleep
   4. Before container.loadPersistentStores start
   5. Before fatalError
   6. At the end of Persistence init function
3. Users update to this version and executed it, I received all the logs except log 5, which means the completionHandler closure was not executed.

So I think I might have triggered a Core Data migration bug and need to find a way to avoid this bug.