c# Regular high HTTP response times with HTTPClientFactory

Question

We have an application that performs HTTP requests to an another in-house microservice. After upgrading from dotnet 7 to dotnet 8, we are seeing the HTTP response times for these HTTP requests spike regularly, from 100ms to around 1 second. These spikes do not occur for every request but do happen every couple of minutes. Below is the HTTP response times from before and after dotnet upgrade:

Here is a closer look at the response times showing the periodic spikes:

Metrics from the called microservice indicate all response were made within a 1 to 3 ms time, indicating the external service is not the source of the performance degradation observed.

The application is hosted on AWS Fargate. No change was made to the CPU and RAM resources assigned to the Fargate tasks. As part of the migration to dotnet 8 we changed the underlying docker image from mcr.microsoft.com/dotnet/aspnet:7.0-alpine3.16 to mcr.microsoft.com/dotnet/aspnet:8.0. When we upgraded from dotnet 6 to 7, we experienced similar behaviour with aspnet:7.0-alpine3.17. Downgrading to aspnet:7.0-alpine3.16 reduced the size of the spikes.

We are using HTTPClientFactory to send the HTTP requests and all other metrics for the application are comparable with what we saw with dotnet 7. It is only the HTTP response times that have deteriorated, which suggests this is where the issue is. Has anyone else experienced similar behaviour? We believe this is the correct way to implement HTTPClientFactory.

 builder.Services.AddHttpClient<IExternalClient, ExternalClient>("ExternalClient");

The ExternalClient HTTP class

public class ExternalClient : IExternalClient
{
    private readonly HttpClient c_httpClient;

    public ExternalClient(
        HttpClient httpClient)
    {
        this.c_httpClient = httpClient;
        this.c_httpClient.Timeout = TimeSpan.FromSeconds(2);
    }


    public async Task<Response> ExecuteAsync(
        ClientURL url,
        Request request)
    {
        const string _mediaTypeHeaderValue = "application/json";

        var _serialisedRequest = JsonConvert.SerializeObject(request);
        var _content = new StringContent(_serialisedRequest);
        _content.Headers.ContentType = new MediaTypeHeaderValue(_mediaTypeHeaderValue);

        var _responseMessage = await this.c_httpClient.PostAsync(url, _content);
        var _stringResult = await _responseMessage.Content.ReadAsStringAsync();

        return JsonConvert.DeserializeObject<Response>(_stringResult);
    }
}

Its called from the controller

return await this.c_externalClient.ExecuteAsync(url, request);

• Are we using HTTPClientFactory correctly?
• Has anyone experienced similar behaviour on dotnet?