I have a lambda with Kafka trigger. Lambda takes a batch of Kafka messages. How to return Kafka messages to the Kafka cluster to retry if the processing of message was failed similarly as SQS, DynamoDBStreams, KinesisDataStreams using aws_lambda_powertools?
Does anybody know how to treat successful messages, and "return" failed Kafka messages to the Kafka cluster? Or is it impossible?
In the documentation it is written that Lambda retries the whole batch of messages: If your function returns an error for any of the messages in a batch, Lambda retries the whole batch of messages until processing succeeds or the messages expire. You can send records that fail all retry attempts to an on-failure destination for later processing.
Then does it lead to the duplication of processing messages? how to handle and ignore already processed messages?