Wait for signal to start generating data from another process in python

Question

I have two independent processes in python: producer and consumer. Producer generates files and consumer does some processing on files.

To test both applications, I find myself constantly starting two programs, and producer has a delay function, etc. which is a pain.

What is the quickest way to implement some kind of signaling machinery in python so that consumer says "GO!" and producer starts doing things it does.

This way I can have producer running all the time.

Answer 1

One simple way to do this (you didn't mention what platform(s) you care about, so I'll assume you want something cross-platform) is a UDP socket with a known port. If the consumer just listens on localhost port 12345, it can block on a sock.recvfrom call every time it needs to wait, and the producer can do a sock.sendto to notify it.

For example, here's a consumer:

#!/usr/bin/env python

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('127.0.0.1', 12345))

while True:
    dummy, addr = sock.recvfrom(1024)
    # work on files until done

And a producer:

#!/usr/bin/env python

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

for thing in things:
    # produce files for consumer
    sock.sendto(b'X', ('127.0.0.1', 12345))

Other things to consider:

• On Unix, there are advantages to using Unix sockets instead of UDP sockets.
• On Windows, there are advantages to using named pipes instead of sockets.
• You may want to make the consumer a daemon (with or without a built-in "service" tool to start and stop it) on Unix or a Windows service on Windows. (You can even merge the service tool into the producer, so, e.g., the default behavior is to start a consumer if one isn't there, then shut it down when it's done, but it can also be used to launch a long-running consumer in the background.)
• You can extend this pretty easily to send more than just an empty notification—e.g., send a different message if you're done producing.