WCF NetTCP with Background Threading

Question

Background:

I have a system that hosts WCF services inside a Windows Service with NetTCP binding. To add a new service to the collection you simply add the standard WCF config entries inside <system.serviceModel -> services /> and then add a line inside a custom configuration section that tells the hosting framework it needs to initialize the service. Each service is initialized with its own background thread and AppDomain instance to keep everything isolated.

Here is an example of how the services are initialized:

Host
  - ServerManager
    - ServiceManager 
      - BaseServerHost

The ServerManager instance has a collection of ServiceManagers that each correlate to a single service instance which is where the standard WCF implementation lies (ServiceHost.Open/Close, etc). The ServiceManager instance instantiates (based on the config - it has the standard assembly/type definition) an instance of the service by use of the BaseServerHost base class (abstract). Every service must inherit from this for the framework to be able to use it. As part of the initialization process BaseServerHost exposes a couple of events, specifically an UnhandledException event that the owning ServiceManager attaches itself to. (This part is critical in relation to the question below.)

This entire process works exceptionally well for us (one instance is running 63 services) as I can bring someone on who doesn't know anything about WCF and they can create services very quickly.

Question:

The problem I have run into is with background threading. A majority of the exposed methods on our endpoints do a significant amount of activity after a standard insert/update/delete method call such as sending messages to other systems. To keep performance up (the front-end is web-based) we let the initial insert/update/delete method do its thing and then fire off a background thread to handle all the stuff an end-user doesn't need to wait for to complete. This option works great until something in that background thread goes unhandled and brings the entire Windows service down, which I understand is by design (and I'm OK with).

Based on all of my research I have found that there is no way to implement a global try/catch (minus using the hacked config of enabling 1.1 handling of background crashing) so my team will have to go back and get those in the appropriate places. That aside, what I've found is on the endpoint side of the WCF hosting appears to be in its own thread on each call and getting that thread to talk to the "parent" has been a nightmare. From the service viewpoint here is the layout:

Endpoint (svc - inherits from BaseServerHost, mentioned above)
  - Business Layer
    - Data Layer

When I catch an exception on a background thread in the business layer I bubble it up to the Endpoint instance (which inherits from BaseServerHost) which then attempts to fire BaseServerHost's UnhandledException event for this particular service (which was attached to by the owning ServiceManager that instantiated it). Unfortunately the event handler is no longer there so it does nothing at all. I've tried numerous things to get this to work and thus far all of my efforts have been in vain.

When looking at the full model (shown below), I need to make the Business layer know about its parent Endpoint (this works) and the endpoint needs to know about the running BaseServerHost instance which needs to know about the ServiceManager that is hosting it so the errors can be bubbled up to this for use in our standard logging procedures.

Host
 - ServerManager
      - ServiceManager <=====================
           - BaseServerHost                ||
                - Endpoint (svc)           ||
                     - Business Layer <======
                          - Data Layer

I've tried static classes with no luck and even went as far as making ServerManager static and expoting its previously internal collection of ServiceManagers (so they can be shutdown), but that collection is always empty or null too.

Thoughts on making this work?

EDIT: After digging a little further I found an example of exactly how I envision this working. In a standard ASP.NET website, on any page/handler etc. you can use the HttpContext.Current property to access the current context for that request. This is exactly how I would want this to work with a "ServiceManager.Current" returning the owning ServiceManager for that service. Perhaps that helps?

Answer 1

Maybe you should look into doing something with CallContext:

http://msdn.microsoft.com/en-us/library/system.runtime.remoting.messaging.callcontext.aspx

You can use either SetData/GetData or LogicalSetData/LogicalGetData, depending on whether you want your ServiceManager to be associated with one physical thread (SetData) or a "logical" thread (LogicalSetData). With LogicalSetData you could make the same ServiceManager instance available within a thread as well as within that thread's "child" threads. Will try to post a couple of potentially useful links later when I can find them.

Here is a link to the "Virtual Singleton Pattern" on codeproject.

Here is a link to "Thread Singleton"

Here is a link to "Ambient Context"

All of these ideas are similar. Essentially, you have an object with a static Current property (can be get or get/set). Current puts its value in (and gets it from) the CallContext using either SetData (to associate the "Current" value with the current thread) or LogicalSetData (to associate the "Current" value with the current thread and to flow the value to any "child" threads).

HttpContext is implemented in a similar fashion.

System.Diagnostics.CorrelationManager is another good example that is implemented in a similar fashion.

I think the Ambient Context article does a pretty good job of explaining what you can accomplish with this idea.

Whenever I dicsuss CallContext, I try to also include this link to this entry from Jeffrey Richter's blog.

Ultimately, I'm not sure if any of this will help you or not. One it would be useful would be if you had a multithreaded server application (maybe each request is fulfilled by a thread and multiple requests can be fulfilled at the same time on different threads), you might have a ServiceManager per thread. In that case, you could have a static Current method on ServiceManager that would always return the correct ServiceManager instance for a particular thread because it stores the ServiceManager in the CallContext. Something like this:

public class ServiceManager
{
  static string serviceManagerSlot = "ServiceManager";

  public static ServiceManager Current
  {
    get
    {
      ServiceManager sm = null;
      object o = CallContext.GetData(serviceManagerSlot);
      if (o == null)
      {
        o = new ServiceManager();
        CallContext.SetData(serviceManagerSlot, o);
      }
      sm = (ServiceManager)o;
      return sm;
    }

    set
    {
      CallContext.SetData(serviceManagerSlot, value);
    }
  }
}

Early in your process, you might configure a ServiceManager for use in the current thread (or current "logical" thread) and then store in the "Current" property:

ServiceManager sm = new ServiceManager(thread specific properties?);
ServiceManager.Current = sm;

Now, whenever you retrieve ServiceManager.Current in your code, it will be the correct ServiceManager for the thread in which you are current executing.

This whole idea might not really be what you want.

From your comment you say that the CallContext data that you try to retrieve in the event of an exception is null. That probably means that exception is being raised and/or caught on a different thread than the thread on which the CallContext data was set. You might try using LogicalSetData to see if that helps.

As I said, I don't know if any of this will help you, but hopefully I have been clear enough (and the examples have also been clear enough) so you can tell if these ideas apply to your situation or not.

Good luck.