Maximum number of concurrent requests in Owin

Question

I self-host a small web application in a console application using OWIN.

Before reaching the ApiController there's a single middleware registered:

public class HealthcheckMiddleware : OwinMiddleware
{
    private readonly string DeepHealthEndpointPath = "/monitoring/deep";
    private readonly string ShallowHealthEndpointPath = "/monitoring/shallow";

    public HealthcheckMiddleware(OwinMiddleware next)
        : base(next)
    {
    }

    public async override Task Invoke(IOwinContext context)
    {
        try
        {
            string requestPath = context.Request.Path.Value.TrimEnd('/');
            if (requestPath.Equals(ShallowHealthEndpointPath, StringComparison.InvariantCultureIgnoreCase)
                || requestPath.Equals(DeepHealthEndpointPath, StringComparison.InvariantCultureIgnoreCase))
            {
                context.Response.StatusCode = (int) HttpStatusCode.OK;
            }
            else
            {
                await Next.Invoke(context);
            }
        }
        catch (Exception ex)
        {
            // This try-catch block is inserted for debugging
        }
    }
}

Here Next.Invoke invokes the controller method, which basically forwards the http request to another API asynchronously, i.e. the main line of interest is:

var response = await _httpClient.SendAsync(outgoingRequest);

However, if I try to submit 10 http requests to the API like this (not awaiting them on purpose as I want to put preassure on the API)

for (int i = 0; i < 10; i++)
{
    var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, "http://localhost:5558/forwarder");
    httpRequestMessage.Content = new StringContent(JsonConvert.SerializeObject(message), Encoding.UTF8, "application/json");
    httpClient.SendAsync(httpRequestMessage);
}

and then immediately afterwards submit 10 more, then I get the following exception in the catch block in the HealthcheckMiddleware:

InvalidOperationException: This operation cannot be performed after the response has been submitted.

Stacktrace:

at System.Net.HttpListenerResponse.set_ContentLength64(Int64 value)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.ResponseHeadersDictionary.Set(String header, String value)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.HeadersDictionaryBase.Set(String key, String[] value)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.HeadersDictionaryBase.set_Item(String key, String[] value)
at Microsoft.Owin.HeaderDictionary.System.Collections.Generic.IDictionary<System.String,System.String[]>.set_Item(String key, String[] value)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.SetHeadersForEmptyResponse(IDictionary`2 headers)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.SendResponseMessageAsync(HttpRequestMessage request, HttpResponseMessage response, IOwinResponse owinResponse, CancellationToken cancellationToken)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at DataRelay.NonGuaranteedDataForwarder.HealthcheckMiddleware.<Invoke>d__3.MoveNext() in C:\_code\DataRelay.NonGuaranteedDataForwarder\HealthcheckMiddleware.cs:line 30

I've tried searching both Stackoverflow and Google, but cannot seem to find anything of value. For instance I found this, but here the developer reads the request after submitting it, which I don't do.

Just in case it could be of interest the full POST method in the ApiController is included here:

    public async Task<HttpResponseMessage> Post(HttpRequestMessage request)
    {
        try
        {
            MetricCollector.RecordIncomingRecommendation();
            using (MetricCollector.TimeForwardingOfRequest())
            {
                string requestContent = await request.Content.ReadAsStringAsync().ConfigureAwait(false);
                var data = JObject.Parse(requestContent);
                string payloadType = data.SelectToken("Headers.PayloadType").ToString();
                Log.Logger.Debug("Received message containing {PayloadType}", payloadType);

                var consumersForPayloadType = _consumers.Where(x => x.DataTypes.Contains(payloadType)).ToList();
                if (consumersForPayloadType.Any())
                {
                    Log.Logger.Debug("{NumberOfConsumers} interested in {PayloadType}",
                        consumersForPayloadType.Count,
                        payloadType);
                }
                else
                {
                    Log.Logger.Warning("No consumers are interested in {PayloadType}", payloadType);
                }

                foreach (var consumer in consumersForPayloadType)
                {
                    try
                    {
                        var outgoingRequest = new HttpRequestMessage(HttpMethod.Post, consumer.Endpoint);
                        outgoingRequest.Content = new StringContent(requestContent, Encoding.UTF8,
                            "application/json");

                        foreach (var header in request.Headers)
                        {
                            if (IsCustomHeader(header, _customHeaders))
                                outgoingRequest.Headers.Add(header.Key, header.Value);
                        }

                        if (!string.IsNullOrWhiteSpace(consumer.ApiKey))
                        {
                            request.Headers.Add("Authorization", "ApiKey " + consumer.ApiKey);
                        }

                        var response = await _httpClient.SendAsync(outgoingRequest);
                        if (!response.IsSuccessStatusCode)
                        {
                            Log.Logger.ForContext("HttpStatusCode", response.StatusCode.ToString())
                                .Error("Failed to forward message containing {PayloadType} to {ConsumerEndpoint}",
                                    payloadType, consumer.Endpoint);
                        }
                    }
                    catch (Exception ex)
                    {
                        MetricCollector.RecordException(ex);
                        Log.Logger.Error(ex,
                            "Failed to forward message containing {PayloadType} to {ConsumerEndpoint}", payloadType,
                            consumer.Endpoint);
                    }
                }

                return request.CreateResponse(HttpStatusCode.OK);
            }
        }
        catch (Exception ex)
        {
            return Request.CreateErrorResponse(HttpStatusCode.ServiceUnavailable, ex);
        }
    }

Answer 1

Try removing .ConfigureAwait(false) everywhere and see if it helps.

E.g. here:

string requestContent = await request.Content.ReadAsStringAsync().ConfigureAwait(false);

---

UPD1: Ok. Check if this exception will occur on server when you use different client for stress testing. For instance this one. Your idea of not awaiting for httpClient.SendAsync(...); is very peculiar.