how to redeliver the message whenever there is any http error code on kafka consumer service (KafkaSource on Knative)

Question

I am using Knative Eventing (Kafka Source, it reads the data from kafka Cluster) to trigger a consumer service, if the event/message did not processed successfully (could be any reason), i want to redeliver the message automatically. I have tried reading through the docs and API reference and it is unclear if this is possible.

Issue: On Knative, once the KafkaSource sends the message to consumer service, the consumer service is processing the request and sending a http error code response (basically message processing failed) and showing as a lag (pending) in the consumer group .

expected: In this scenario want to redeliver the same message before process the next sequential request. is it possible from the Kafka Source to redliver the mssage or Consumer Service redeliver the failed requests ?

Thanks much appreciated!!

Srinivas P.

Answer 1

At the moment Knative Eventing sources doesn't support retrying event delivery. In order to get event deliveries you have to send the event to a Broker or a Channel and configure the delivery properties, as described here:

• Delivery spec: https://github.com/knative/eventing/blob/master/pkg/apis/duck/v1/delivery_types.go#L29
• Broker delivery configuration: https://knative.dev/docs/eventing/broker/
• Channel Subscription delivery: https://knative.dev/docs/eventing/event-delivery/

For example, you can send the events from the KafkaSource to a KafkaChannel and then you can register your subscribers to that KafkaChannel, defining in the Subscription spec the delivery options.

Note: With this approach, you're going to lose message ordering.

Answer 2

Update

Since Knative v0.24 Event Sources have retries built-in.

They retry any error or a status code other than 2xx.

In addition, Kafka Source respects ordered delivery.

Ordered delivery is a per-partition blocking consumer that waits for a successful response from the CloudEvent subscriber before it delivers the next message of the partition.