WCF, BasicHttpBinding: Stop new connections but allow existing connections to continue

Question

.NET 3.5, VS2008, WCF service using BasicHttpBinding

I have a WCF service hosted in a Windows service. When the Windows service shuts down, due to upgrades, scheduled maintenance, etc, I need to gracefully shut down my WCF service. The WCF service has methods that can take up to several seconds to complete, and typical volume is 2-5 method calls per second. I need to shut down the WCF service in a way that allows any previously call methods to complete, while denying any new calls. In this manner, I can reach a quiet state in ~ 5-10 seconds and then complete the shutdown cycle of my Windows service.

Calling ServiceHost.Close seems like the right approach, but it closes client connections right away, without waiting for any methods in progress to complete. My WCF service completes its method, but there is no one to send the response to, because the client has already been disconnected. This is the solution suggested by this question.

Here is the sequence of events:

1. Client calls method on service, using the VS generated proxy class
2. Service begins execution of service method
3. Service receives a request to shut down
4. Service calls ServiceHost.Close (or BeginClose)
5. Client is disconnected, and receives a System.ServiceModel.CommunicationException
6. Service completes service method.
7. Eventually service detects it has no more work to do (through application logic) and terminates.

What I need is for the client connections to be kept open so the clients know that their service methods completed sucessfully. Right now they just get a closed connection and don't know if the service method completed successfully or not. Prior to using WCF, I was using sockets and was able to do this by controlling the Socket directly. (ie stop the Accept loop while still doing Receive and Send)

It is important that the host HTTP port is closed so that the upstream firewall can direct traffic to another host system, but existing connections are left open to allow the existing method calls to complete.

Is there a way to accomplish this in WCF?

Things I have tried:

1. ServiceHost.Close() - closes clients right away
2. ServiceHost.ChannelDispatchers - call Listener.Close() on each - doesn't seem to do anything
3. ServiceHost.ChannelDispatchers - call CloseInput() on each - closes clients right away
4. Override ServiceHost.OnClosing() - lets me delay the Close until I decide it is ok to close, but new connections are allowed during this time
5. Remove the endpoint using the technique described here. This wipes out everything.
6. Running a network sniffer to observe ServiceHost.Close(). The host just closes the connection, no response is sent.

Thanks

Edit: Unfortunately I cannot implement an application-level advisory response that the system is shutting down, because the clients in the field are already deployed. (I only control the service, not the clients)

Edit: I used the Redgate Reflector to look at Microsoft's implementation of ServiceHost.Close. Unfortunately, it calls some internal helper classes that my code can't access.

Edit: I haven't found the complete solution I was looking for, but Benjamin's suggestion to use the IMessageDispatchInspector to reject requests prior to entering the service method came closest.

Answer 1

Guessing:

Have you tried to grab the binding at runtime (from the endpoints), cast it to BasicHttpBinding and (re)define the properties there?

Best guesses from me:

• OpenTimeout
• MaxReceivedMessageSize
• ReaderQuotas

Those can be set at runtime according to the documentation and seem to allow the desired behaviour (blocking new clients). This wouldn't help with the "upstream firewall/load balancer needs to reroute" part though.

Last guess: Can you (the documention says yes, but I'm not sure what the consequences are) redefine the address of the endpoint(s) to a localhost address on demand? This might work as a "Port close" for the firewall host as well, if it doesn't kill of all clients anyway..

Edit: While playing with the suggestions above and a limited test I started playing with a message inspector/behavior combination that looks promising for now:

public class WCFFilter : IServiceBehavior, IDispatchMessageInspector {
    private readonly object blockLock = new object();
    private bool blockCalls = false;

    public bool BlockRequests {
        get {
            lock (blockLock) {
                return blockCalls;
            }
        }
        set {
            lock (blockLock) {
                blockCalls = !blockCalls;
            }   
        }

    }

    public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {          
    }

    public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) {         
    }

    public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
        foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) {
            foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) {
                endpointDispatcher.DispatchRuntime.MessageInspectors.Add(this);
            }
        } 
    }

    public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) {
        lock (blockLock) {
            if (blockCalls)
                request.Close();
        }
        return null;
    }

    public void BeforeSendReply(ref Message reply, object correlationState) {           
    }
}

Forget about the crappy lock usage etc., but using this with a very simple WCF test (returning a random number with a Thread.Sleep inside) like this:

var sh = new ServiceHost(new WCFTestService(), baseAdresses);

var filter = new WCFFilter();
sh.Description.Behaviors.Add(filter);

and later flipping the BlockRequests property I get the following behavior (again: This is of course a very, very simplified example, but I hope it might work for you anyway):

// I spawn 3 threads Requesting a number..
Requesting a number..
Requesting a number..
// Server side log for one incoming request
Incoming request for a number.
// Main loop flips the "block everything" bool
Blocking access from here on.
// 3 more clients after that, for good measure
Requesting a number..
Requesting a number..
Requesting a number..
// First request (with server side log, see above) completes sucessfully
Received 1569129641
// All other messages never made it to the server yet and die with a fault
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request before the block.
Error in client request before the block.

Answer 2

I haven't implemented this myself, so YMMV, but I believe what you're looking to do is pause the service prior to fully stopping it. Pausing can be used to refuse new connections while completing existing requests.

In .NET it appears the approach to pausing the service is to use the ServiceController.

Answer 3

Maybe you should set the

ServiceBehaviorAttribute and the OperationBehavior attribute. Check this on MSDN

Answer 4

In addition to the answer from Matthew Steeples.

Most serious load balancers like a F5 etc. have a mechanism to identify if a node is alive. In your case it seems to check whether a certain port is open. But alternative ways can be configured easily.

So you could expose e.g. two services: the real service that serves requests, and a monitoring "heart beat"-like service. When transitioning into maintenance mode, you could first take the monitoring service offline which will take the load away from the node and only shutdown the real service after all requests finished processing. Sounds a bit weird but might help in your scenario...

Answer 5

Does this WCF Service authenticate the user in any way? Do you have any "Handshake" method?

Answer 6

Is there an api for the upstream firewall? The way we do this in our application is to stop new requests coming in at the load balancer level, and then when all of the requests have finished processing we can restart the servers and services.

Answer 7

I think you might need to write your own implementation with a helper class that keeps track of all running requests, then when a shutdown is requested, you can find out if anything is still running, delay shutdown based on that... (using a timer maybe?)

Not sure about blocking further incoming requests... you should have a global variable that tells your application whether a shutdown was requested and so you could deny further requests ...

Hope this may help you.

Answer 8

My suggestion is to set an EventHandler when your service goes into a "stopping state", use the OnStop method. Set the EventHandler indicating that your service is going into a stopping state.

Your normal service loop should check if this event is set, if it is, return a "Service is stopping message" to the calling client, and do not allow it to enter your normal routine.

While you still have active processes running, let it finish, before the OnStop method moves on to killing the WCF host (ServiceHost.Close).

Another way is to keep track of the active calls by implementing your own reference counter. you will then know when you can stop the Service Host, once the reference counter hits zero, and by implementing the above check for when the stop event has been initiated.

Hope this helps.