Does MSMQ get corrupted?

Question

I just started with a company which has a home grown cloud solution. They have built their central queuing mechanism around SQL Server. When talking to the technical director, he tells me that they have tried MSMQs in the past, but ran into issues with high volume queues getting corrupted. I have been doing a search on the internet, but cannot find anything about this issue.

Has anyone heard of such a thing? Are there any good articles on this?

Also, they are not using WCF for this purpose. Would the transactional nature of WCF solve these corruption issues?

Answer 1

No, MSMQ does not get corrupted.

What Hugh is referring to is the memory mapping of storage files into virtual memory and the fragmentation of such.

MSMQ works in 4MB blocks, regardless of the message size. Send a 5kb message - get a 4MB block. Send another 5kb message - use the same block. Eventually the block fills up and MSMQ starts a fresh one. When all the messages in a block are read and consumed (as opposed to just peeked), the file is deleted. MSMQ waits several hours before deleting, just in case a new message arrives and needs a home - no point creating a new file when there's one hanging around empty.

If the messages are of the same size, the blocks are easily filled and emptied as they arrive and depart.

If the message are of variable size, and are not read FIFO, then holes can start appearing. For example, a 5kb message is read and a 4 kb message arrives, leaving 1kb of unusable space. Over time, the amount of allocated memory will deviate from the actual volume of messages due to accumulated dead space.

This is all normal. There is no defrag for MSMQ.

Notes

1. Disk writes in 4kb chunks
2. Messages require contiguous space
3. Writing to disk

Answer 2

I have had a similar experience, though corrupted is not the term I would have used. "Fragmented" is better. My experience is limited to using transactional queues so I am unsure if you can avoid this problem by using non-trasactional queues.

When subjected to heavy load the msmq service can exhibit a memory-leak behavior, where it simply will never release the memory. So you can end up with msmq running at higher and higher working memory footprints, which will impact performance.

I don't really understand the underlying cause, but it has to do with the way MSMQ stores messages on disk, and these storage files become fragmented.

Just because I have experienced this does not necessarily mean that you will too, it may be something specific to the way we had msmq set up.

Note this is not the same phenomena as disk fragmentation.

Read more about it here, where I also have outlined a workaround here.

Regarding WCF I assume you're referring to the msmq bindings for WCF. I have used both the bindings available for msmq and don't believe this would alter this behavior as it's to do with the underlying msmq subsystem.