How does Azure WebJob QueueTrigger message processing works in case of distributed instances

Question

Azure WebJob (created using Azure WebJob SDK) with QueueTrigger is configured to process messages added to 'queue1'. https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-get-started

This webjob will have multiple instances (e.g. 3 instances) of it running simultaneously and all of them poll for messages sent to the same queue 'queue1'.

Is it possible for more than 1 instance to dequeue the same message at the same time? If yes, is it our webjob's responsibility to have distributed locking mechanism to allow message processing only once (if that's the goal)?

Answer 1

How does Azure WebJob QueueTrigger message processing works in case of distributed instances

• Multiple instances of the webjobs run in parallel.
• Message gets added to the queue.
• Message will be handled by any of the available instances once it gets added to the queue.
• If any of the message processing get failed, it is again added back to the queue and picked by another available instance.

Azure WebJobs uses a new feature lease to ensure each meassage is processed only by one instance at a time.

The WebJobs SDK supports distributed processing of message from a queue.

Is it possible for more than 1 instance to dequeue the same message at the same time?

Yes, when more than 1 instance of WebJobs are running there are changes to process the same message from the queue. To avoid this, we can use SingletonAttribute as mentioned in the MSDoc.

The Singleton attribute ensures that only one instance of a function runs, even when there are multiple instances of the host web app. The Singleton attribute uses distributed locking to ensure that one instance runs.

The WebJob has to implement the distributed locking mechanism.

As mentioned in Multiple instances

If your web app runs on multiple instances, a continuous WebJob runs on each machine, and each machine will wait for triggers and attempt to run functions. The WebJobs SDK queue trigger prevents a function from processing a queue message multiple times. To ensure that only one instance of a function runs even when there are multiple instances of the host web app, we can use the Singleton attribute