Azure service bus deadletter queue message unable to stay in deadletter for longer time

Question

I have Azure Main function app which processes coming messages from service bus.

If exception occurs because of API is down then I push message into Dead Letter Queue.

I have another azure Dead function app which continuously read DLQ and push back message into main queue.

And main Function app check if api is down then push message again to DLQ this should go on until API is up and Main function app process message successfully.

But issue is after some time message purge automatically.

I check Default Message Time To Live is 2 hours 30 minutes but message unable to stay there so long. I think in Dead Function app when I push message back to Main. in main queue message may be identified as duplicate but not sure.

What should I do to sustain message into DLQ for given Default Message Time To Live?

Dead letter function app code which push back message to main queue -

 MessageSender sender = new MessageSender(Environment.GetEnvironmentVariable("ConnectionStringSettingName"), UtilityHelper.GetAndValidateAppSetting("TopicName"), RetryPolicy.Default);

            Message deadLetterMessage = new Message(message.Body);

            foreach (KeyValuePair<string, object> userProperty in message.UserProperties)
            {
                deadLetterMessage.UserProperties.Add(userProperty.Key, userProperty.Value);
            }

            //Send the message to the Active Queue
            await sender.SendAsync(deadLetterMessage);

Main Function App Code-

[FunctionName("ProcessMessage")]
        public static async System.Threading.Tasks.Task RunAsync([ServiceBusTrigger("mytopic", "mysub", Connection = "ConnectionStringSettingName")]Message mySbMsg, string lockToken, MessageReceiver messageReceiver, ILogger log)
        {
            
            try
            { 
                  log.LogInformation("Process message...");
            }
            catch (Exception ex)
            {
                log.LogInformation($"Send message to dead-letter: {ex}");
                await messageReceiver.DeadLetterAsync(lockToken);
            }
        }

My logic is: until API is up I'm sending message into DLQ and DLQ function app pushing it back to main queue so that I don't lost message.

Update

Find out root cause:

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full ErrorCode: NoBufferSpaceAvailable ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was fullat Microsoft.Azure.ServiceBus.ServiceBusConnection.CreateConnectionAsync(TimeSpan timeout)

How to tackle this?

Answer 1

The Default Message Time To Live setting does not apply to dead-letter. As per documentation:

There's no automatic cleanup of the DLQ. Messages remain in the DLQ until you explicitly retrieve them from the DLQ and call Complete() on the dead-letter message.

The code above for the function that returns dead-lettered messages to the original queue will complete each dead-lettered message and those will be gone from the dead-letter queue.