C# - HttpWebRequest / GetResponse performance

Question

I have a HTTPS based API which I need to call a lot of times. Using HttpWebRequest.Create(uri).GetResponse() takes from 50ms up to 500ms or more to execute. In order to check the response time I implemented it like this:

private void Action()
{
    WebRequest request = HttpWebRequest.Create("https://.....");
    using (WebResponse response = request.GetResponse()) { }
}

and then calling it:

private long getTime()
{
    Stopwatch sw = new Stopwatch();
    sw.Start();
    Action();
    return sw.ElapsedMilliseconds;
}

Output for several calls:

Time: 746 ms
Time: 51 ms
Time: 50 ms
Time: 50 ms
Time: 51 ms
Time: 49 ms
Time: 2417 ms ???
Time: 52 ms
Time: 52 ms
Time: 51 ms
Time: 50 ms
Time: 201 ms
Time: 234 ms
Time: 204 ms
Time: 51 ms
Time: 50 ms
Time: 50 ms
Time: 52 ms
Time: 280 ms
Time: 264 ms

First QUESTION: I was wondering if there was any way to speed up GetResponse in order for it to take as less time as possible?

Now.. because I need to make A LOT of different requests with different URLs, in order to speed up the process I decided to use TPL Dataflow Block (instead of Parallel.Foreach), since Parallel.Foreach is mainly used for CPU bound work, and what I am doing is I/O bound (there's also processing of the response, so also a bit of CPU work). And when I use TPL Dataflow Block the processing of 250 URLs takes up to 7 seconds to execute, whereas Parallel Foreach takes 15 seconds or more, so I'm positive TPL Dataflow Block usage is the right way to go. How I implemented it:

//CALL:
var block = new ActionBlock<string>(uri => Action(uri), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 200 });
foreach (var URL in urlArray)
{
    block.Post(URL);
}
block.Complete();
await block.Completion;

//Action(uri):
private void Action(string uri)
{
    WebRequest request = HttpWebRequest.Create(uri);
    using (WebResponse response = request.GetResponse()) { }
}

And since I am not happy with 7s execution I've tried tweaking ServicePointManagerin order to speed it up, things I've tried so far and NONE of it worked:

ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
ServicePointManager.SetTcpKeepAlive(false, 0, 0);
ServicePointManager.DefaultConnectionLimit = 1000;

Second QUESTION: If it is not possible speeding up GetResponse() in order to achieve faster execution, is there any way to tweak TPL Dataflow Block into having better performance?

EDIT: My goal is to execute all calls AS FAST AS possible.

Answer 1

You could speedup your solution by using GetResponseAsync. See also this Micorsoft walkthrough where both methods (synchronous and asynchronous) are explained in depth.