I have requirement to implement microservice, where it consumes kafka events for millions of users with time-to-live of 10 minutes, stores them in some Database like Redis and when a user connect to the microservice, it needs to provide all user specific events and to the connected user. It also need to provide all real-time events as long as the user is connected to the service.
These events has time-to-live for 10 minutes and event structure is like this:
{ "event":"propChn", "refId":"01HCFRNFXH8VR90MAE21M0E7NQ" }
I am considering Redis-Streams for this use-case. I am new to Redis-streams and currently I am thinking to create stream per user, so that I can add user events to user specific stream using xadd and then retrieve them using xread, whenever user connect to the microservice. However I also have requirement to read all real-time events as long as user is connected to my microservice. There could be millions of users connected to microservice instances.
Is there any efficient way to retrieve real time events for all connected users? I read about redis consumer group, is there any limit on number of consumer groups? lets say 10.000 users connected to service, so 10.000 consumer groups waiting for events from respective user specific stream.
Note: I am using vertx-redis-client library.