Hang in COM application with C# plugin

Question

I've got a problem where our application hangs on our customers' machines that I've been on for days now without solving. The problem arises quite randomly from what we've seen, even though that may not be the true case. The customer also reports that the CPU is peaking when the application hangs.

The problem is that I've no idea where the application is failing (hanging). We have a couple of plugins written in C# as plugins for the main COM application as well.

I've managed to get the customer to take a MiniDump with ProcExp on a machine where the problem occurred. However, I'm not very familiar with WinDBG or MiniDumps either for that matter. I've run !analyze -v and !analyze -v -hang, which produces some output, including the stack below. For what I'm able to tell, it seems the application is transitioning into one of our C# plugins (CLR) and then back again to COM, which ought to be an interface back to the main application that is available to the plugins. But then what? Is it possible to say something more from this stack?

The main application is written in VB6 if that matters.

0012da70 7e3693e9 7e3693a8 0012daf0 00000000 ntdll!KiFastSystemCallRet
0012da9c 7e37a43b 0012daf0 00000000 0000000f user32!NtUserPeekMessage+0xc
0012dac8 7348e6fd 0012daf0 00000000 0000000f user32!PeekMessageA+0xeb
0012db1c 77532a9c 00d03774 00000000 00000000 msvbvm60!CMsgFilter::MessagePending+0x21f
0012db60 77532a48 00000102 0012dbec 00000001 ole32!CCliModalLoop::HandlePendingMessage+0x40
0012dba8 7751eda6 80010116 80010115 00000000 ole32!CCliModalLoop::HandleWakeForMsg+0x46
0012dbbc 77547297 0012ddf0 00000001 0012dbe8 ole32!CCliModalLoop::BlockFn+0x8b
0012dc30 0ae8a2fd 00000002 00000001 00000001 ole32!CoWaitForMultipleHandles+0xcf
0012dc50 0ae8a264 00000000 00000001 00000001 mscorwks!NT5WaitRoutine+0x51
0012dcbc 0ae8a1c8 00000001 0012ddf0 00000000 mscorwks!MsgWaitHelper+0xa5
0012dcdc 0af3ccd0 00000001 0012ddf0 00000000 mscorwks!Thread::DoAppropriateAptStateWait+0x28
0012dd60 0af3cd65 00000001 0012ddf0 00000000 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
0012ddb0 0ae8c026 00000001 0012ddf0 00000000 mscorwks!Thread::DoAppropriateWait+0x40
0012de00 0ae90f4c 00000001 05ae5c80 0012de60 mscorwks!Thread::JoinEx+0x86
0012de10 0b00ea40 00000001 00000001 5bdf0a2d mscorwks!Thread::Join+0x14
0012de60 0af0b9a7 00000001 0012deb0 0af0ba10 mscorwks!RCWCleanupList::CleanupWrappersInCurrentCtxThread+0x15a
0012de6c 0af0ba10 001df674 0012df10 5bdf0afd mscorwks!RCW::Initialize+0x78
0012deb0 0af0b6a7 001df674 0012df10 5bdf0b6d mscorwks!RCW::CreateRCW+0x84
0012df20 0af0b7b5 00000000 0012df5c 5bdf0b21 mscorwks!COMInterfaceMarshaler::CreateObjectRef+0x45
0012df6c 0ae892a8 5bdf3065 0012e6c8 0012e6c8 mscorwks!COMInterfaceMarshaler::FindOrCreateObjectRef+0xac
0012e428 0ae89444 001df658 11a11014 00000001 mscorwks!GetObjectRefFromComIP+0x1ec
0012e448 0ae894ab 05b3b0a8 001df658 0e896204 mscorwks!UnmarshalObjectFromInterface+0x19
0012e464 0af164d1 0012e6c8 0012e6ac 0af164c1 mscorwks!InterfaceMarshalerBase::ConvertSpaceNativeToCLR+0x30
0012e470 0af164c1 0012e748 0012e738 5bdf32e1 mscorwks!DefaultMarshalOverrides<InterfaceMarshalerBase>::ReturnCLRFromNativeRetval+0xb
0012e6ac 0aeb4bff 11741a76 0012e734 0012e74c mscorwks!RunML+0x9ac
0012e780 11740172 05ae5c80 0012e7d4 501b6046 mscorwks!CLRToCOMWorker+0x25f
...
... Stack begins down here in our main COM application.

Edit 1: Some thoughts that I've got after reading some other forum posts. It seems to me that the hanging is introduced following a marshaling of data types from .NET to COM. Now I've read about another problem which seemed to originate in the fact that local variables passed into a COM method were garbage collected and thus the VB6 run time had problems dealing with the deallocated memory. That other post was not exactly this problem, but it made me think.

In the .NET code calling into COM we have this type of code

void SomeNETMethod(MyObject obj, bool doIt) {
    string someString = obj.SomeString;
    this.myComComponent.DoItInCOM(ref someString, ref doIt); // This is where it hangs.
}

Could the ref bool as well as the directly from method to method passed parameter have anything to do with anything? As you can see, I'm stumbling in the dark here...

Answer 1

Review your "DoItInCOM" method as the kernel dump indicates a modal dialog is being displayed (likely as a result of an error within the COM method call.)

The .NET parameters are correctly being marshalled over to the COM side. If you were using some non-standard C# type you may run into a problem with serialization but this is not the case here.

If you place an "On Error Goto" handler in the VB code you can likely suppress errors that would generate a message on the UI thread. If the error is lower level (the VB runtime is throwing it) you will need to resolve that problem with fixes to the COM component directly. Also review the Windows Event Viewer for VB Runtime origintatin messages for further information.