Azure Service Bus: Best way to implement exponential retry policy for failed to process messages

Question

I am continuously receiving messages in peek mode and abandoning them if processing fails (Not the delivery). However, the message immediately becomes available again and is received for processing again. It fails quickly again and after max deliveries it is dead-lettered.

Is there a way to configure the topic/subscription to wait before releasing the message after it is abandoned? Preferably in exponential manner.

Of course I am open for suggestions via the code as well.

Answer 1

There is not a way to set an exponential back-off in the Service Bus configuration. I've experienced the same issue, and have done the following:

1. Dequeue the message, and marked the message as received.
2. Perform processing within a try/catch block. If there is an exception, enqueue a new message with a scheduled delivery at a certain point in the future.

We've wrapped our Service Bus Message Queue payload in a class that specifies the number of delivery attempts. We multiply the number of delivery attempts times a constant, then add that number to the current dateTime for scheduled delivery in the future. After the number of delivery attempts that we want to try are exceeded, we explicitly dead letter the message.

Edit 7-17-2020 Consider using Azure Durable Task Framework which has a customizable retry policy built right in.

Answer 2

Another option is using MassTransit which has support for Azure Service Bus.

Take a look at its extensive retry configuration.

Note that MassTransit is effectively doing the retries in memory after the message has been received so you'll need to adjust your topic subscription's MaxDeliveryCount and MaxAutoRenewDuration settings appropriately.

Your configuration might look something like:

var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
    var host = cfg.Host(serviceBusConnectionString, hst => { });

    cfg.UseMessageRetry(retryConfigurator =>
        RetryConfigurationExtensions.Exponential(retryConfigurator, ...);

    cfg.SubscriptionEndpoint(
        "subscriptionName",
        "topicPath",
        e =>
        {
            e.Consumer<SomeConsumer>();
            // Let MassTransit do the retries
            e.MaxDeliveryCount = 1;
            e.MaxAutoRenewDuration = TimeSpan.FromMinutes(10);
        });
});

Answer 3

You can use Scheduled Messages for this.

Essentially, when you need to retry a message you just schedule it in the future and Service Bus will add it to the queue again once enough time has passed:

        ServiceBusSender queue = ...
        int secs = // calculate the delay

        var msg = new ServiceBusMessage("My scheduled message body")
        {
            ApplicationProperties =
            {
                ["RetryCount"] = retryCount,
            },
        };

        logger.LogInformation("Scheduling for " + secs + " secs");
        await queue.ScheduleMessageAsync(msg, DateTimeOffset.Now.AddSeconds(secs));

You must add information about retry count in a header or the body. Otherwise you won't know how many times you have tried it and cannot really calculate the future date.

Answer 4

The message should remain unavailable/hidden for the duration of the message lock. If you abandon the message explicitly (using the AbandonAsync method), then the lock is cancelled, making the message available for consumption; if you instead finish your process without explicitly abandoning the message, Azure will wait until the lock duration is finished to make the message available again, this could give you a bit more of delay. You can tune this to implement your retry operation; you can also implement more complete approaches (like exponential backoff) as explained here

Answer 5

I am looking into this topic as well and I came across the class RetryExponential class from Microsoft.

RetryExponential Class

^{Namespace: Microsoft.ServiceBus
Assembly: Microsoft.ServiceBus.dll}

Represents an implementation of a retry policy. For each time the messaging operation must be retried, the delay between retries grows in a staggered, exponential manner.

public sealed class RetryExponential : Microsoft.ServiceBus.RetryPolicy