Looking into building a web app that runs on MVC4 at AppHarbor. In the interest of responsiveness and performance, slightly longer running tasks (typically, generating/sending emails, resizing images, payment transaction processing etc) would be handled by putting a message on a message queue.
There would also be one or more workers, somewhere out there, dequeueing messages and processing whatever needs to be processed as a result. The central queueing mechanism would be RabbitMQ, specifically the hosted CloudAMQP service available through AppHarbor. In theory, this architecture would allow "infinite" scalability by adding more workers.
Now, in the interest of good architecture, testability etc., I want to put RabbitMQ behind one or more interfaces that can easily be mocked. I have a few considerations to take into account when defining these interfaces.
- Until I get any paying users, I am limited to the free CloudAMQP offering. That means a maximum of three simultaneous connections.
- Given the previous constraint, I want to try to limit myself to one connection per application. One for the MVC app, one per worker. That's it.
- Connections in RabbitMQ are designed to be living for a long time. Still, many examples show code that explicitly opens a connection (then a channel), sending a message, then closing it again.
- For multithreading, one can use the same connection, but not the same channel. As far as I understand, it is ok to share a connection in a multithreaded app, then open several channels, for instance one per thread.
- My MVC app would typically be a publisher only. The worker app would be both a consumer and a publisher - for instance, the worker could receive a message to process a payment, which would result in it publishing a message to invoke an email message to the user indicating success or failure.
- For the MVC app, connections would normally be used only for a very short time - for queueing a message.
- For the workers, I'm thinking long-running connections, more or less permanently open for the lifetime of the worker.
Ok, great, so that's a lot of stuff. Not having any experience with RabbitMQ before this project, I am haunted by some additional bullet points.
- Interface segregation principle. Should I divide this into two separate interfaces - like IQueueServiceConsumer and IQueueServiceProducer - or is that taking things too far? I find I can always use SRP etc. to divide things further into more atomic units, and I don't want Uncle Bob hunting me down (for more than the I in the interface names), but I am wondering how far I have to take this.
- Given that I will only have one single webserver, at least at first, would it be an idea to actually have a long-living connection to the queue in the web app as well? Opening and closing them gives a theoretical chance of several happening at once, meaning conflicts and potential messages lost. Messages lost is not acceptable.
- How would I handle the lifetime of the connection in this case? Open it inside my Ninject module (where my interface is actually bound to the RabbitMQ-specific implementation), and disconnect it somehow when the application is... recycled? How could I do something like that?
- For the workers, life seems easier. Put a method on the interface to open the connection, then serve channels to threads for getting messages. Make sure the application plays nice by calling CloseConnection as appropriate. Or implement IDisposable.
- There is a chance - however small - that RabbitMQ might be replaced by something else in the future, thus I don't want my interfaces to be too specific to that particular service bus (in the sense that this is how I am using it) implementation.
Did anyone else already work through the same challenges when building a similar application? Do you have any advice for the architecture and real-life implementation of an interface to the message queue? Am I just being neurotic about this or are my worries worth acknowledging?
If it matters, my current messaging needs are covered by one-way messages, no need for any response besides acknowledging received messages once processing is complete.
Thanks for any insights!
Masstransit will do all of this for you. Either use MT or look at what they have done