HttpClient, WebClient or HttpWebRequest takes 90 seconds to read a file on the first attempt, after that it runs quickly

Question

I have an application which runs on a PC and downloads files from a web server running in an embedded device which is connected to the PC using RNDIS over USB. The RNDIS connection looks like a network adaptor and lets me run TCP/IP connections between the PC and the embedded device.

When the device is powered up I can use a web browser to view content on the device within a few seconds. But when my app runs, the first call to HttpClient takes 90 seconds to complete. After that. HttpClient runs normally. If I close my app and open it again then HttpClient now works correctly on the first attempt. The 90 second delay when the app starts is very noticeable to users.

This problem seems to only occur on some PCs and not on others. I have tried using HttpClient, WebClient and HttpWebRequest but the results are the same. I have also experimented with the settings in ServicePointManager and ServicePoint after reading others topics on StackOverflow, but they have no effect no this issue.

I can demonstrate the problem using just these few lines of code in a console app:

static async Task Main(string[] args)
{
    using (HttpClient httpClient = new HttpClient())
    {
        string url = "http://169.254.21.151/eventlog/2020-10-11.txt";
        for (; ; )
        {
            DateTime t1 = DateTime.Now;
            try
            {
                HttpResponseMessage httpResponseMessage = await httpClient.GetAsync(url);
                DateTime t2 = DateTime.Now;
                string content = await httpResponseMessage.Content.ReadAsStringAsync();
                DateTime t3 = DateTime.Now;
                TimeSpan dt1 = t2 - t1;
                TimeSpan dt2 = t3 - t2;
                Console.WriteLine($"Http Read {content.Length} bytes after {dt1.TotalMilliseconds:F0} ms + {dt2.TotalMilliseconds:F0} ms");
            }
            catch (HttpRequestException)
            {
                DateTime t2 = DateTime.Now;
                TimeSpan dt = t2 - t1;
                Console.WriteLine($"Http Failed to read after {dt.TotalMilliseconds:F0} ms");
            }
            await Task.Delay(1000);
        }
    }
}

I tested the code above like this:

1. Power up embedded device
2. Use a web browser to test HTTP connection and download a file
3. Run the code above

The results show that the first call to httpClient.GetAsync(url) takes 90 seconds to complete. 90 seconds strikes me as being a suspicious number which suggests there is a timeout setting somewhere that I haven’t found yet. The 90 second measured delay is very consistent across multiple tests.

Http Read 1106623 bytes after 90449 ms + 0 ms
Http Read 1106623 bytes after 1080 ms + 0 ms
Http Read 1106623 bytes after 1070 ms + 0 ms
Http Read 1106623 bytes after 1110 ms + 0 ms

Is there a setting somewhere that I am missing?

Answer 1

I found a brute force solution. I think the problem is somewhere in the ServicePointManager, so my solution is to avoid using ServicePointManager when making HTTP requests to this hardware by using TcpClient instead of HttpClient and then building the HTTP request headers myself. I know this comes with a performance penalty due to creating and destroying sockets for each request, but I think that is acceptable in this case as the volume of requests to the device is fairly small. I would welcome a suggestion of a more elegant solution.

My code which is equivalent to the code in the question is:

static void Main(string[] args)
{
    string host = "169.254.21.151";
    string file = "/eventlog/2020-10-11.txt";
    for (; ; )
    {
        DateTime t1 = DateTime.Now;
        try
        {
            string query = $"GET {file} HTTP/1.1\r\nConnection: close\r\nHost: {host}\r\n\r\n";
            byte[] queryBytes = Encoding.UTF8.GetBytes(query);
            using (TcpClient tcpClient = new TcpClient(host, 80))
            {
                tcpClient.GetStream().Write(queryBytes, 0, queryBytes.Length);
                using (StreamReader streamReader = new StreamReader(tcpClient.GetStream()))
                {
                    string response = streamReader.ReadToEnd();
                    int index = response.IndexOf("\r\n\r\n");
                    if (index > 0)
                    {
                        string headers = response.Substring(0, index);
                        string content = response.Substring(index + 4);
                        if (headers.Contains("HTTP/1.1 200 OK"))
                        {
                            DateTime t2 = DateTime.Now;
                            TimeSpan dt = t2 - t1;
                            Console.WriteLine($"Http Read {content.Length} bytes after {dt.TotalMilliseconds:F0} ms");
                        }
                    }
                }
            }
        }
        catch (Exception)
        {
            DateTime t2 = DateTime.Now;
            TimeSpan dt = t2 - t1;
            Console.WriteLine($"Http Failed to read after {dt.TotalMilliseconds:F0} ms");
        }
        Thread.Sleep(1000);
    }
}