Dequeue a collection to write to disk until no more items left in collection

Question

So, I have a list of file shares. I then need to obtain ALL folders in these file shares. This is all the "easy" stuff I have done. Ultimately after some logic, I am adding an object into a collection.

All the while this is happening in the background using Async/Await and Tasks, I want to be able to have another thread/task spin up so it can keep going through the collection and write data to disk.

Now, for each folder, I obtain security information about that folder. There will be at LEAST 1 item. But for each item, I add this information into a collection (for each folder).

I want to write to disk in a background until there are no more folders to iterate through and the job is complete.

I was thinking of using a BlockingCollection however this code does smell and ultimately does not close the file because of the while(true) statement.

private static BlockingCollection<DirectorySecurityInformation> AllSecurityItemsToWrite = new BlockingCollection<DirectorySecurityInformation>();

if (sharesResults.Count > 0)
{
    WriteCSVHeader();

    // setup a background task which will dequeue items to write.
    var csvBGTask = Task.Run(async () =>
    {
        using (var sw = new StreamWriter(FileName, true))
        {
            sw.AutoFlush = true;
            while (true)
            {
                var dsi = AllSecurityItemsToWrite.Take();
                await sw.WriteLineAsync("... blah blah blah...");
                await sw.FlushAsync();
            }
        }
    });
    allTasks.Add(csvBGTask);
}

foreach(var currentShare in AllShares)
{
   var dirs = Directory.EnumerateDirectories(currentShare .FullName, "*", SearchOption.AllDirectories);
   foreach(var currentDir in dirs) { // Spin up a task in the BG and run to do some security analysis and add to the AllSecurityItemsToWrite collection }
}

This is at its simplest but core example. Any ideas? I just want to keep adding on the background task and have another task just dequeue and write to disk until there are no more shares to go through (shareResults).

Answer 1

Recommand to use Channel.

Channel<DirectorySecurityInformation> ch = 
    Channel.CreateUnbounded<DirectorySecurityInformation>();

Write

var w = ch.Writer;
foreach(var dsi in DSIs)
    w.TryWrite(dsi);
w.TryComplete();

Read

public async void ReadTask()
{
    var r = ch.Reader;
    using (var sw = new StreamWriter(filename, true))
    {
        await foreach(var dsi in r.ReadAllAsync())
            sw.WriteLine(dsi);
    }
}

Answer 2

while (true)
{
    var dsi = AllSecurityItemsToWrite.Take();
    //...
}

Instead of using the Take method, it's generally more convenient to consume a BlockingCollection<T> with the GetConsumingEnumerable method:

foreach (var dsi in AllSecurityItemsToWrite.GetConsumingEnumerable())
{
    //...
}

This way the loop will stop automatically when the CompleteAdding method is called, and the collection is empty.

But I agree with shingo that the BlockingCollection<T> is not the correct tool in this case, because your workers are running on an asynchronous context. A Channel<T> should be preferable, because it can be consumed without blocking a thread.