Automatically expire Service Fabric Reliable Dictionary objects via RunAsync

Question

I'm trying to add automated deletions to expired reliable dictionary objects and it looks like I have to implement my own way according to this: https://stackoverflow.com/a/36466890/7293543

My approach was to use the "RunAsync" task and have it constantly running a while loop. It worked for the first few times but I'm getting a weird error once I added a few more objects to my dictionary. Is this a stupid approach? How are other people automatically clearing their reliable dictionary objects?

protected override async Task RunAsync(CancellationToken cancellationToken)
{
    while (true)
    {
        cancellationToken.ThrowIfCancellationRequested();
        await deleteExpired("MyDictionaryName", cancellationToken);
    }
}

private async Task deleteExpired(string dictionaryName, CancellationToken cancellationToken)
{
    var myDictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<string, CacheObject>>(dictionaryName);
    Dictionary<string, CacheObject> ret = new Dictionary<string, CacheObject>();

    using (var tx = StateManager.CreateTransaction())
    {
        var count = myDictionary.GetCountAsync(tx);

        if (count.Result > 0)
        {
            IAsyncEnumerator<KeyValuePair<string, CacheObject>> e = (await myDictionary.CreateEnumerableAsync(tx)).GetAsyncEnumerator();

            while (await e.MoveNextAsync(cancellationToken))
            {
                if (e.Current.Value.expiration <= DateTime.Now)
                {
                    await myDictionary.TryRemoveAsync(tx, e.Current.Key);
                    await tx.CommitAsync();
                    ServiceEventSource.Current.ServiceMessage(this.Context, String.Format("Object deleted at {0} - key: {1} expired at {2}", DateTime.Now.ToString(), e.Current.Key, e.Current.Value.expiration.ToString()));
                }
            }
        }
    }
}

Error: The error occurs on "while(true)" after I added a few reliable dictionary objects into my dictionary.

Managed Debugging Assistant 'FatalExecutionEngineError' has detected a problem in 'C:\SfDevCluster\Data_App_Node_2\CacheApplicationType_App339\CachePkg.Code.1.0.0\Cache.exe'.

Additional information: The runtime has encountered a fatal error. The address of the error was at 0x8c46ed90, on thread 0x1980. The error code is 0x80131623. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.

Answer 1

If the Reliable Dictionary contains more than one item ready to be expired, the above code keeps using the transaction after it is terminated (committed in the above case). This causes an assert in the enumeration since it is being used after the transaction it was bound to has terminated.

I will update the recommendation in Reliable Collections to

• Do not use an enumeration after the transaction used to create is terminated (committed/aborted) or disposed.