WCF not handle 1000 call per second

Question

I am working on a WCF Service that is hosted in Windows Service, using nettcpbinding.

when i tried to perform load test on the service i built a simple client that call the service about 1000 call in second, the return from the service take about 2 to 8 seconds at first and after leaving the simple client running for about half hour the time to return the result increased, and some client gives some time out exceptions for the send time which was configured to be 2 minutes.

i revised the service throltting configuration and it's like this

these are the steps i tried to perform:

1. revised the service throttling configuration

   <serviceThrottling maxConcurrentCalls="2147483647" maxConcurrentInstances="2147483647" maxConcurrentSessions="2147483647"/>

2. was working on Windows 7 machine, so i moved to server 2008 but the same result.

3. update the configuration of tcp binding to be like the following NetTcpBinding baseBinding = new NetTcpBinding(SecurityMode.None, true); baseBinding.MaxBufferSize = int.MaxValue;

           baseBinding.MaxConnections = int.MaxValue;
           baseBinding.ListenBacklog = int.MaxValue;
           baseBinding.MaxBufferPoolSize = long.MaxValue;
   
           baseBinding.TransferMode = TransferMode.Buffered;
           baseBinding.MaxReceivedMessageSize = int.MaxValue;
           baseBinding.PortSharingEnabled = true;
           baseBinding.ReaderQuotas.MaxDepth = int.MaxValue;
           baseBinding.ReaderQuotas.MaxStringContentLength = int.MaxValue;
           baseBinding.ReaderQuotas.MaxArrayLength = int.MaxValue;
           baseBinding.ReaderQuotas.MaxBytesPerRead = int.MaxValue;
           baseBinding.ReaderQuotas.MaxNameTableCharCount = int.MaxValue;
           baseBinding.ReliableSession.Enabled = true;
           baseBinding.ReliableSession.Ordered = true;
           baseBinding.ReliableSession.InactivityTimeout = new TimeSpan(23, 23, 59, 59);
   
   
           BindingElementCollection elements = baseBinding.CreateBindingElements();
           ReliableSessionBindingElement reliableSessionElement = elements.Find<ReliableSessionBindingElement>(); 
           if (reliableSessionElement != null)
           {
               reliableSessionElement.MaxPendingChannels = 128;
   
   
   
               TcpTransportBindingElement transport = elements.Find<TcpTransportBindingElement>();
   
               transport.ConnectionPoolSettings.MaxOutboundConnectionsPerEndpoint = 1000;
   
               CustomBinding newBinding = new CustomBinding(elements);                    
               newBinding.CloseTimeout = new TimeSpan(0,20,9);
               newBinding.OpenTimeout = new TimeSpan(0,25,0);
               newBinding.ReceiveTimeout = new TimeSpan(23,23,59,59);
               newBinding.SendTimeout = new TimeSpan(0,20,0);
               newBinding.Name = "netTcpServiceBinding";
   
               return newBinding;
           }
           else
           {
               throw new Exception("the base binding does not " +
                   "have ReliableSessionBindingElement");
           }
   

4. changed my services function to use async and await

   public async Task<ReturnObj> Connect(ClientInfo clientInfo)
   {
       var task = Task.Factory.StartNew(() =>
       {
           // do the needed work
           // insert into database
           // query some table to return information to client
       });
   
   
       var res = await task;
       return res;
   }
   

   and updated the client to use async and await in it's call to the service.

5. applied the Worker thread solution proposed in this link https://support.microsoft.com/en-us/kb/2538826 although i am using .net 4.5.1, and set the MinThreads to 1000 worker and 1000 IOCP

after all this the service start to handle more requests but the delay still exist and the simple client take about 4 hours to give time out

the strange thing that i found the service handle about 8 to 16 call within 100 ms, regarding the number of threads currently a live in the service.

i found a lot of articles talk about configuration needed to be placed in machine.config and Aspnet.config, i think this is not related to my case as i am using nettcp on windows service not IIS, but i have implemented these changes and found no change in the results.

could some one point me to what i am missing or i want from the service something it can't support?

Answer 1

This is most likely due to the concurrency mode set to Single (this is default value). Try setting ConcurrencyMode to Multiple by adding ServiceBehaviourAttribute to your service implementation.

Be sure to check documenttation: https://msdn.microsoft.com/en-us/library/system.servicemodel.concurrencymode(v=vs.110).aspx

Example:

// With ConcurrencyMode.Multiple, threads can call an operation at any time.   
// It is your responsibility to guard your state with locks. If 
// you always guarantee you leave state consistent when you leave 
// the lock, you can assume it is valid when you enter the lock.

[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
class MultipleCachingHttpFetcher : IContract

You may be interested also in Sessions, Instancing, and Concurrency article which describes concurrency problems.

Answer 2

It could be how your test client is written. With NetTcp, when you create a channel, it tries to get one from the idle connection pool. If it's empty, then it opens a new socket connection. When you close a client channel, it's returned back to the idle connection pool. The default size of the idle connection pool is 10, which means once there are 10 connections in the idle pool, any subsequent closes will actually close the TCP socket. If your test code is creating and disposing of channels quickly, you could be discarding connections in the pool. You could then be hitting a problem of too many sockets in the TIME_WAIT state.
Here is a blog post describing how to modifying the pooling behavior.