Python producer and consumer system design

149 views Asked by At

If I'm writing a Python program that execute other two executables(C++) in a producer and consumer manner, what is the right lib to do? asyncio? multiprocessing?

The main execution thread or process is:

  1. monitoring status of producer and consumer processes.
  2. if one of them fails, terminate the other.
  3. if Ctrl-C is received, terminate both of them
  4. if both of them finishes, exit and report success.

The Producer:

  1. Keep generating data to a temp file. (that temp file might be a fifo from os.mkfifo()?)

The Consumer:

  1. Keep consuming the temp file

Both Producer and Consumer are CPU intensive, and they need to run in parallel. Ideally the intermediate temp file doesn't need to touch real storage and stay in memory. Otherwise performance is a big concern since the temp file is of size 10GB or so.

0

There are 0 answers