Signalling a producer task from a consumer task when working with a BlockingCollection

425 views Asked by At

I have a pretty basic application that uses a Producer task and a Consumer task to work with files. It is based off the example here http://msdn.microsoft.com/en-us/library/dd267312.aspx

The basics of the program is that the Producer task enumerates the files on my hard drive and calculates their hash values and does a few other things. Once the Producer has finished working with a file, it Enques the file and the Consumer then grabs it.

The Consumer task has to connect to a remote server and attempt to upload the file. However, if the Consumer encounters an error, such as, not being able to connect to the remote server I need it to signal the Producer task that it should stop what it is doing and terminate. If the server is down, or goes down, there is no need for the Producer to continue cycling through thousands of files.

I have seen plenty of samples of signalling the Consumer task from the Producer task by using .CompleteAdding() on the BlockingCollection object but I am lost as to how to send a signal to the Producer from the Consumer that it should stop producing.

1

There are 1 answers

2
Martin James On

You could use a return queue. If one of the items generates an error/exception, you could load it up with error data and queue it back to the producer. The producer should TryTake() from the return queue just before generating a new item and handle any returned item appropriately. This beats using some atomic boolean by enabling the item to signal back with extended error information that could be used to decide what action to take - the producer may not always want/need to stop. Also, you could then queue up errored items to a GUI list and/or logger.

It's tempting to say that the consumer should return items anyway, whether they are errored or not, so that they can be re-used insted of creating new ones all the time. This, however, intruduces latency in detecting/acting on errors unless you use two return queues to prioritize error returns.

Oh - another thing - using the above design, if it has to stop, the producer could retain errored items in a local queue an re-issue one occasionally. If the server comes back up, (as indicated by the return of a successful item), the producer could re-issue the errored jobs from the local queue again before generating any more new ones. With care, this could make your upload system resilient to server reboots.