Core-Data background save performance issues

Question

I have written an application using core data that collects various data from a sensor over wifi. My data thread runs in the background reading data off a socket from wifi and creating new core-data entities. The problem is I'm getting about 27 updates a second and when I've tried to have the thread call a save on the objects as soon as it receives them my UI starts to lag and the program becomes unusable I'm not sure if this is due to a design flaw in my code or related to the nature of how Core Data works.

I'd like to know some options for how I can actively do saves in the background without impacting my UI or any other application code. I had thought perhaps there was a way to fire off a batch save of say 500 records every few seconds in another background thread or something but I wasn't sure exactly a) how to implement this and b) if it was possible.

I'm actively creating objects with calls to:

[NSEntityDescription insertNewObjectForEntityForName:@"RPYL" inManagedObjectContext:managedObjectContext];

And once I'm done with the data collection I make a call to:

[managedObjectContext save:&error]

Answer 1

You cannot save in the background with no effect on your main thread — the SQLite developers are of the antiquated opinion that "Threads are evil. Avoid them.". There is therefore a lot of mutual exclusion involved in using SQLite. While it is saving, the persistent store is locked, regardless of where the save originates from. If anything else needs to access the store during that time then it must wait. If you have affected indexes then the save involves taking every item in the affected tables and sorting them according to the indexed column as SQLite implements indexing by binary search. So the cost can be a function of how much you're inserting plus how much you already have in the store.

First of all try intelligent faulting when doing the thread confinement hop over onto your the main queue — use NSFetchRequest and explicitly say you want things pre-faulted. Then you'll be able to access them without any further trips to the store, saving you from hitting the mutex. Try running with -com.apple.CoreData.SQLDebug 1 set for your target to see quite how often you are going to the store: if it's not something you've optimised for then I guarantee it'll be a huge amount more than you thought.

General tips also apply: schedule against the runloop, not directly onto the main queue, if you have work you want to do but not while the user is interacting with the app. The runloop switches into and out of tracking mode so you can just schedule your work for the default mode and it'll automatically avoid running while the UI is busy under user control.

If that doesn't resolve the problem and your records are intended to be immutable upon receipt with no complicated queries then consider using Core Data as the permanent store but an immutable non-managed object of your own design for runtime use. Build those in the background and pass them forwards.

EDIT: batching your saves is easy enough — just literally don't call save too frequently. One solution is to have a BOOL indicating whether a save is scheduled; when you want to save then if one isn't scheduled, schedule one for a second from now with dispatch_after. If one is scheduled then do nothing. If you've data constantly coming in then more complicated schemes (eg, when did I last save, so how long until I can save again?) aren't going to gain you much.

Answer 2

There are a few things you have to watch out for and a few things you can do to improve performance.

Setup: I use a DataController class that sets up and provides access to all the Core Data objects (persistentStoreCoordinator etc). Inside that class I have a class method that I use to run Core Data operations in the background. It creates a fresh NSManagedObjectContext (because we should never pass an NSManagedObjectContext between threads) and takes a block to be executed in that new context. The method looks like this:

+ (void)saveDataInBackgroundWithSaveBlock:(void(^)(NSManagedObjectContext *context))saveBlock   
                          completionBlock:(void(^)(void))completionBlock
{     
    // nested contexts are broken on iOS 5, see http://stackoverflow.com/questions/11786436/core-data-nested-managed-object-contexts-and-frequent-deadlocks-freezes
    // that's why we directly use the persistentStoreCoordinator instead

    NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
    // setting the undoManager to nil dramatically improves performance and memory usage
    context.undoManager = nil;
    [context setMergePolicy:NSMergeByPropertyObjectTrumpMergePolicy];
    [context setPersistentStoreCoordinator:[[self class] sharedInstance].persistentStoreCoordinator];        

    // make sure our changes are merged into our main `NSManagedObjectContext`
    NSNotificationCenter *notificationCenter = [NSNotificationCenter defaultCenter];
    [notificationCenter addObserver:[[[self class] sharedInstance] managedObjectContext]
                           selector:@selector(mergeChangesFromContextDidSaveNotification:)
                               name:NSManagedObjectContextDidSaveNotification
                             object:context];

    [context performBlock:^{
        saveBlock(context);
        if ([context hasChanges])
        {
            NSError *error;
            if ([context save:&error]) {
                NSLog(@"saving in bg successful");
            } else {
                NSLog(@"Error bg save: %@", error);
            }
        }

        dispatch_async(dispatch_get_main_queue(), ^{
            completionBlock();
        });
    }];
}

You then call this method like so:

[DataController saveDataInBackgroundWithSaveBlock:^(NSManagedObjectContext *context) {
    // create NSManagedObjects is the background with the given context
} completionBlock:^{
    // do something on the main thread, for example
    // [self.collectionView reloadData];
}];

Performance: Setting context.undoManager = nil; helps a lot with memory usage and performance. Also, don't save after every object you create. Save periodically instead, that will help as well.

Credit for a lot of this goes to http://www.cimgf.com/2011/05/04/core-data-and-threads-without-the-headache/.