I'm trying to implement a multi-agent system in C++. To maximize the number of agents per PC, I've thought of the following high-level design:
Each agent will be represented as an instance of a class. (In a multi-agent system, an agent is an independent/autonomous entity).
Each instantiated agent will be spawned off as a thread.
If the OS (Windows or Linux) allows a maximum of A processes, and each process allows spawning of a maximum of B threads, then the total number of agents would be A x B.
The agents will communicate with each other using a publisher/subscriber message broker. Plan on using RabbitMQ.
For the results to be statistically valid, the requirement is to have a few million agents (e.g. say, 5 million).
Ideally, I'd like each agent to be implemented as one process, instead of one thread. Reasons being, avoiding shared memory bottleneck and interdependency among agents. E.g. A specific thread-crash or a process-crash could cause the entire thread pool to crash with it. An agent represented as a process would not have such limitations.
However, as I understand, the upper limit on the number of processes in an OS is limited (possibly few thousand processes)? If my understanding is correct, the requirement of a few million agents will not work with a process-only strategy.
Question:
What would be the best possible strategy (if at all) to implement such a multi-agent system with multi-million agents on a PC server (say, with an i7 CPU). Essentially, what would be the best implementation strategy that maximizes the number of agents per CPU, without the constraints of shared memory and interdependency?
Thanks in advance.