I have a data processing workflow based on Luigi which uses a central scheduler (so not the in-memory version). I run multiple tasks in parallel. Luigi starts worker processes for those tasks by using multiprocessing.Process:start(). The code of my actual tasks is executed in those worker processes whose management I don't control.
My problem is the following: My tasks need access to a shared resource, i.e. a sequence number generator. Creating those numbers upfront and making them parameters of my tasks is not an option. The resource needs to be accessed at the moment the task is really executed, not at scheduling time.
A possible solution would be to spawn an additional TCP service and let all worker processes talk to that service. But this is quite heavy weight and adds complexity.
I looked at all kind of synchronization primitives and shared memory, but could find a proper solution. The code needs to run only on Linux.
I wonder if there is build in Python or Linux functionality / library, that could generate an incremental number sequence over multiple processes, which is simpler than creating a dedicated service.