We have Kafka Streams execution microservice which gets the records from a topic (source node) and then transforms that record into a payload and calls REST API post call so that the payload is submitted to a REST API server is not maintained by us it is external to us and we have no control. But clients will configure the REST API urls and we need to call this REST API so that payload is submitted to their servers so they receive the notification of payload.
Problem is we are having so much latency with these REST API calls as they dont scale well in Kafka Streams application.
One way to solve is, instead directly calling REST API in kafka streams, write the transformed payload into some DB. And then use a scehduler to poll the DB and do the REST API call with proper task status etc.
Is there any other better way to solve this problem.