C# MongoDb.driver How to insert many records without duplicates and return results

Question

I would like to ask for an advice, how add many records without duplicates and return a results.

My case: I have a List, have to insert all of them to the database but need to exclude records that are already in db(comparison need to be done based on UniqueId field which is in object list - UniqueId is a single index). I need to know which objects was duplicated and which was successfully added to the db. Currently, I have a working solution, but the performance isn't best because I'm inserting documents one by one and based on MongoWriteException=11000(duplicate) I'm able to recognize if the document is duplicated. I'm thinking about a different approach using InsertMany, I should also be able to catch duplicates(have to parse errors message and retrieve UniqueId which is duplicated), but what's about records that was successfully processed. Initially I though that I can see in List on ID property that is assigned by mongo but ID is populated for all entities(added and duplicated) so then have to iterate through list and check ids.

Do you have any better idea/solution how to solve such case?

Answer 1

you can take advantage of the bulkwrite functionality like this:

var list = new[] {
    new Book { Id = "1", UniqueId = "100", Title = "one" }, //existing record
    new Book { Id = "2", UniqueId = "200", Title = "two" },
    new Book { Id = "3", UniqueId = "300", Title = "three" }
};

var models = new List<WriteModel<Book>>(list.Length);

foreach (var book in list)
{
    var upsert = new ReplaceOneModel<Book>(
        filter: Builders<Book>.Filter.Eq(e => e.UniqueId, book.UniqueId),
        replacement: book)
    { IsUpsert = true };

    models.Add(upsert);
}

var result = await collection.BulkWriteAsync(models);

var addedIds = result.Upserts.Select(u => u.Id.AsString);

var replacedIds = list.Select(x => x.Id).Except(addedIds);

Console.WriteLine("added ids: " + string.Join(",", addedIds));
Console.WriteLine("duplicate ids: " + string.Join(",", replacedIds));
Console.ReadLine();

keep in mind this will actually replace the existing record if any of the properties/fields have changed. will be ignored if the data is exactly the same in the entity and the document in the db.