I'm working with a "firm" real-time system (the result will be unuseful if served after the deadline) which requires maximum throughput and robustness to load.
More details about our system, we are processing telecommunication messages. The messages will go through several step with different concerns. However the messages order is very important. The steps are like these:
- Query something from Redis in-memory database
- Query something from another relational in-memory database
- Some business logic computional processing
- Update data into Oracle RDBMS
- Send messages to other server through Sockets
We are currently setup a system with a distributor and 8 processing units (which runs through the messages sequentially) on only 1 server. We decided to maximize the performance per unit first, before doing horizontal scale.
I've tried several approaches (i'm only passing the messages through the first two steps and some data collecting logic):
- Proactor pattern: by implementing the event-loop and high-level asynchronous API with callbacks by myself (using Java API). However, the performance is not as expected, the messages are not in order.
- SEDA: the overall performance is good when i'm using a thread-pool for each stage. However, to preserve the message order, I tried using only a thread per stage, the performance is even worse than the Proactor pattern.
- LMAX Disruptor: the "default" batching characteristic brings both the latency and throughput down because I find it a bit hard to apply batching to our current business logic. When I avoid using batching, the performance is ok now, however, it's only about 1.5x faster than the sequential approach (even i'm using a thread for each handler, so it's 3 thread)
- A "naive" multithreading approach: using a BlockingQueue + some worker threads, the performance is better than I expected. 1.75-2X faster than the LMAX Disruptor approach. I find it a little bit counter-logic here...
So, have i done anything wrong with my approaches above? Does the event-driven architecture really fit our need for a high throughput, firm real-time system?
P/S: It's my first time asking a question on Stackoverflow, so please give me advices if there're any problems with my questions :D. Thanks :D.