WCF service hosted on Azure App service never seems to finish threads opened for processing

Question

I have deployed a WCF service to Azure App Service that performs just one task - send a message to the topic. Although app works fine with normal load, it starts experiencing higher thread count as soon as load on the app increases. The app instance becomes unhealthy when the threads count limit is reached.

Those threads stay in waiting state forever. We tried scaleout option on thread count metrics but the app just keeps on adding more instances as the earlier instance still had almost all threads waiting and remain unhealthy forever.

This is performed in the below sequence.

1. Accept a request.
2. initialize a Service bus topic client
3. Send the requested message to the topic.
4. Closed the topic client.

While sending a burst of 1000 requests, the app works but the number of threads initiated always stays in the waiting state. However, while these threads are waiting CPU stays at 0%. The average response time from this service is also under 100 ms avg.

After sending 1000 requests to this service, I see a similar number of threads open.

What could be the potential root cause of this issue? Is there any issue with my code to send the message to the topic?

  public async Task SendAsync(Message message)
        {
            try
            {              
                await _topicClient.SendAsync(message);
            }
            catch(Exception exc)
            {
                throw new Exception(exc.Message);
            }
            finally
            {
                await _topicClient.CloseAsync();
            }
        }

enter image description here

Answer 1

Thanks for responding. These are good suggestions and I will look to review my implementation inline with these.

The good news is that I was able to resolve the issue, it wasn't related to the topic client as I thought earlier. It was due to how I was registering dependency injection.

I am implementing a WCF service based on .Net Framework 4.8 and initially, we did not include Global.asax but registered DI in the service controller constructor. The implementation worked till we realized (as part of performance testing) it seems to add additional threads when we added ILogger dependency. Those additional threads never cool down but were adding up as the service received more requests.

To resolve, I moved DI registration into Application_Start in global.asax.

Answer 2

The code sample you provided does not really tell us much. We do not know how SendAsync(Message message) is being invoked. Is your image your queue count that drops to 0 before accepting more messages? I'm assuming a client calls your WCF app service which tells it send the message to service bus?

It does sound like you are hitting the 1000 maximum connections. Your _topicClinet should be a singleton for your app domain that all clients use. You also should only need one app service instance if all you're doing is message forwarding. No need for scaling unless there's more processing that you haven't alluded to.

Have a look at the Service Bus messaging best practices doc for more suggestions.