Message Response Zombies occurring with errors Codes 0xC0C01B4C and 0xc0c016b5 no Orchestration

Question

Consider the following message flow in BizTalk.

We have several MLLP receive ports/locations setup receiving HL7v2 messages in one application. These ports each receive slightly different message types.

Let's call this one RP1

In another application we have send ports that subscribe to each respective receive port. These send ports each have an outbound map that transforms the messages in HL7v3 and submits it to a WCF (request/response) service.

Let's call this SP1

The WCF service then processes and validates the HL7v3 and sends back an HL7v3 ack message. The SP1 send port has custom send and receive pipeline components. The receive (from the WCF response) just takes the message and promotes certain fields which will be used later for subscriptions.

There are then two more Send ports. SP2 that subscribes to positive ACKs. SP3 to the Negative based on the fields promoted above. The positive ACKs are just consumed and the negative ACKs are emailed off to support staff.

The issue is that in about 10% of messages we are seeing 1 of these 2 error messages popping up:

A response message for two-way receive port "SP.CDX.LAB_MICRO.SubmitCDA.WCFCustom" is being suspended as the messaging engine could not correlate the response to an existing request message. This usually happens when the host process has been recycled.
MessageId:  {731623F3-995B-4C57-BD21-12865AD78717}
InstanceID: {084BD473-C857-4C5E-A49B-8A86EA2CAC39}
The following stored procedure call failed: " { call [dbo].[bts_UpdateMsgbox_BizTalkServerReceive]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}". SQL Server returned error string: "The statement has been terminated.;Cannot insert duplicate key row in object 'dbo.InstancesSuspended' with unique index 'IX_InstancesSuspended_InstanceID'. The duplicate key value is (084bd473-c857-4c5e-a49b-8a86ea2cac39, afa466c7-3bd2-4cde-a293-3df3fb5d8543)."

Usually followed by a suspended service instance in the Group viewer:

 The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended

The Service Name of the suspended instance is that of RP1. The Message type of the non-consumed message is that of the ACK from SP1 (so it's the WCF response). This is weird because in my mind RP1 should never be expecting this response message AND there are send ports (SP2, SP3) subscribed to the response message types.

The other point I forgot to make is, there are 3 Receive Ports like RP1, each with 3 Receive Locations and 3 Send Ports subscribing to their respective Receive Ports.

The BizTalk Server is installed across 2 physical servers sharing 1 BizTalkMgmtDb/Messagebox

Previous to this we had the same number of messages feeding in, but it was consolidated (at the sending end) to one single Receive Location. The old solution had multiple Orchestrations, but never experienced this issue.

So why now are the WCF (HL7v3) response messages getting lost and being suspended under an instance of RP1 (HL7v2)?

Here's a basic image of what it looks like.

Answer 1

I was able to fix this by changing the value of the RouteDirectToTP value in the Context Properties of the WCF two-response message. Based on information in these posts:

https://jeremiedevillard.wordpress.com/2010/01/11/how-work-request-response-pattern-part-3/

https://bveldhoen.wordpress.com/2010/09/05/messaging-only-request-response-correlation/

Apparently by default, BizTalk tries to route an ACK back to the original receive port by default. I just did this in my pipeline component:

 msgReceived.Context.Promote("RouteDirectToTP", "http://schemas.microsoft.com/BizTalk/2003/system-properties", false);

and so far the problem has stopped. No more Zombies and no more errors regarding the stored procedure failing.

Answer 2

MLLP Receive Adapter Processing

Acknowledgments with the two-way MLLP receive adapter

When a two-way MLLP receive adapter receives a message, Microsoft BizTalk Accelerator for HL7 (BTAHL7) can generate the following types of ACKs:

• HL7 Enhanced Commit ACK: In this scenario, BTAHL7 sends a Commit ACK on the same connection. It sends out an Application Accept ACK on a different send port.

• Application Accept ACK: In this scenario, BTAHL7 sends an Application Accept ACK on the same connection.

• Static ACK: In this scenario, BTAHL7 sends an ACK on the same connection.

• The type of ACK generated depends on the BTAHL7 Configuration Explorer settings for the party sending the message. The value in fields MSH 15 and 16 of an individual message can override this setting. However, for applications expecting Static ACKs, the configuration can only be set through BTAHL7 Configuration Explorer.

From reading that I would think you would need to configure your adapter to HL7 Enhanced Commit ACK