We are working on a project to build a distributed message and notification system like Gmail /whatsapp.
Currently i am having some scaling and performance issues. Our architecture dropping messages in a kafa message broker. The message payload has the message body and the recipient / groups. The listener then inspects the message and then makes an entry into database against the recipient .
This activity of creating inserting millions of records in the database is not efficient and time cosuming and piles up based on the messages present in the broker. This results in non-regular delivery time periods of the messages.
Can anyone direct me on how to address this issue.
i am always intrigued on how gmail /whatsapp architecture is, as to how soon they are able to deliver their messages to 100s of millions of users instantaneously.