I am trying to model my problem using NATS and Jetstream. Previously, I was using a plain MQTT broker to move messages between my services but we will be requiring temporal decoupling the consumers from the producers. The scenario goes like this:
- we have multiple service producing events on an
activity-log/{entity}/{operation} - those events are consumed by at least another service that persists the message onto a database to provide querying capabilities on all the logs. This service is, hence, subscribed to
activity-log.#messages - the service that persists the messages is containerized and running with multiple instances. We use MQTT5 shared subscriptions (similar to Core NATS queue groups) to "load-balance" handling of those messages
- additionally, there may be other consumers (or none) of the same message, performing a completely different action. Those services are subscribed to a subset of all the activity logs and each extra subscriber will filter the messages by the topic
- those consumers are also "load-balanced". That is the message will be handling by one out of several replicas
- those ex
What would be the best way to model my problem?
My initial idea was to have a stream for this topic activity-log.>.
For that stream have a pull consumer consuming all the messages (the equivalent of the multiple replicas subscribed to activity-log/#) and then other pull consumers with topic filtering.
However that does not seem to be possible because those consumers topics cannot overlap.
I can see the example from the documentation to be similar to what I want to accomplish but push consumers are used. I heard push consumers are not the way to go forward (can't say why, though).