How to trace overhead added by Application Request Routing?

Question

We have a Delphi SOAP service which needs to be SSL-enabled. I opted to use an IIS ARR reverse proxy to do SSL offloading for ease of configuration (compared to OpenSSL and manual certificate + passphrase management). ARR works, but it adds an insane amount of overhead... Response time went from under 2 seconds to 19 seconds for 18 service requests (about 60Kb compressed total).

I added timestamp logging to client & server for when messages are both sent and received. It shows about 1 second added to each request routing through ARR between sending from the client and receipt by the service. The response is routed back very quickly, only the request routing via ARR is slow (see image below).

How can I trace the source of overhead? Is ARR not suited to this use case? I tried tweaking and disabling most settings, including caching. I tried different hosts with clean IIS setups, including a production Windows Server 2012. SSL itself is not the overhead, just having an ARR HTTP reverse proxy causes the delay.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:8987/{R:1}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Request & response samples from Fiddler:

• Raw HTTP Request
• Raw HTTP Response

Answer 1

We have the same problem. I found the root, it's in System.Net.Sockets.Socket.DoConnect The problem is related to IPv6:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/203b6230-e4c0-477c-9a0a-0c21a7ad1615/strange-onesecond-delay-with-tcpconnections-to-localhost?forum=clr

http://msdn.microsoft.com/en-us/library/115ytk56.aspx

"If IPv6 is enabled and the TcpClient(String, Int32) method is called to connect to a host that resolves to both IPv6 and IPv4 addresses, the connection to the IPv6 address will be attempted first before the IPv4 address. This may have the effect of delaying the time to establish the connection if the host is not listening on the IPv6 address."

To resolve it for loopback requests you need to disable IPv6 on a machine, see p.4-5-6: https://stackoverflow.com/a/12403731

Answer 2

Turning off IPv6, as Dmitry suggested, solved this for me.

You could also use 127.0.0.1 in your rewrite, instead of localhost, to force IPv4.

Answer 3

I would say you have something screwy in your setup. We currently run ARR for SSL offloading and conducted throughput testing at significant volume and ARR showed little to no affect on throughput.

I would agree with Brock and say check your App Pool settings first. Basically an ARR App Pool should be set to never recycle.

I would recommend watching episodes 32-38 starting at: http://dotnetslackers.com/articles/iis/Bindings-and-Rules-for-Application-Request-Routing-ARR-Week-32.aspx

Answer 4

My advice: use IIS for the apps, use Apache HTTP Daemon for the proxying.

In the past I've used various pieces of software and hardware for SSL offloading (starting somewhere in 2003 I think). Each with their own pricing level and features. The last years I've switched to solely using Apache HTTP Daemon for that purpose. Even in combination with IIS and on Windows. Apache is easy to configure once you have a running sample and more easily grows to more complex scenarios with forwarding and renaming.

Some instructions how to use Apache HTTP Daemon on Windows as SSL offloading engine can be found on http://www.invantive.com/about-invantive/news/entryid/897/ssl-offloading-for-apache-tomcat.