Exception The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue in Azure

Question

My application is subscribed to an Azure service bus topic. I do some processing when the service receives a message, It works as expected when the processing time is small. But when it takes much time, my services start receiving the same message multiple times.

Exception Type: Azure.Messaging.ServiceBus.ServiceBusException

Exception Message: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . azure(MessageLockLost)

the value of MaxAutoRenewDuration is 2 in this application. So I think if it takes more time it again gets the same message. The prefetchCount value is 0 in the application. I have read some articles and understood that If I increase the value of MaxAutoRenewDuration, It may solve the problem. But again I would like to know the pros and cons of it.

I have one interesting observation as well, I have added a test subscription on this topic, But this test subscription still shows only one message for this message, But my receiving application is receiving multiple messages, so how does it work?

Answer 1

There are two settings you need to consider in your scenario.

1. MaxAutoRenewDuration
2. LockDuration

LockDuration is defined on the entity you receive from. It should alsway be less than MaxAutoRenewDuration. So if processing takes up to 5 minutes but can spike to 7, you should set LockDuration to the max 5 minutes and MaxAutoRenewDuration to something greater than 7.

Answer 2

For Service Bus, each message is locked by the service when a consumer reads and that consumer has exclusive access to the message until either they settle the message (complete, abandon, dead-letter, etc) or until the lock is held long enough that it expires.

To allow for processing that may go on longer than the lock duration but that also may be variable enough that you don't want to increase the duration for all consumers, the ServiceBusProcessor will automatically renew the lock for messages that have been read while the processing handler is still active for those messages.

The MaxAutoRenewDuration option controls how long the processor will renew the message lock for you while your processing takes place. The general recommendation for this value is "long enough that it covers the longest time processing should take", with the intent being to ensure that any delays when processing the message such as retries or slow downstream systems don't cause failures.

The trade-off that you're making is how long this consumer will continue to have exclusive access in the case where your processing hangs or behaves unexpectedly. A shorter MaxAutoRenewDuration will allow the lock to expire and another consumer (possibly the same instance) to read the message again so that processing is not delayed. That comes with the cost that if processing runs longer than this time, the lock may be lost before it is finished and you may process that message again.

In your scenario, it sounds like your processing can sometimes take longer than that duration and the lock is expiring, and it would be helpful to increase unless your application is sensitive to latency and would prefer to process multiple times to reduce delays.

I have one interesting observation as well, I have added a test subscription on this topic, But this test subscription still shows only one message for this message, But my receiving application is receiving multiple messages, so how does it work?

There's not enough context to answer that authoritatively. How you're determining that the "subscription shows only one message"? What does "receiving multiple messages" mean - that you're doing batch receives, that you're seeing this message replay, or that you're seeing multiple messages dispatched for processing?