I searched Stackoverflow and Google but couldn't find an answer for my situation. Here is my design.
1) I have two state machine workflows defined (Parent and Child). The parent WWF oversees the child WWF for status and the child WWF is the one that actually performs the task for a given message.
2) The parent WWF spawns child WWFs (instance.Start()) 500 of them. Then using the replicator activity control (parallel execution mode) within the parent WWF, the child start event is triggered.
3) The code is hosted in a VM (8 GB RAM, 4 CPU cores, IIS 7.0 with its own application pool).
I have noticed 2 different behaviors of the execution of the same code:
A) There are times the execution takes just 8 seconds to finish the processing of all 500 children WWFs which is an excellent.
B) However there are times after doing Instance.Start() in the parent WWF for all 500 children, the replicator activity simply hangs for 30-40s doing nothing and then each child WWF gets processed almost sequentially. So it takes about 300-400s to finish the same job that was completed in 8 seconds previously.
Based on my findings, I think it all comes to Thread pool management. Since my code is NOT directly dealing with the ThreadPool.QueueWorkItem() and WWF actually works on a different thread behind the scene, I have little or no control about the thread management for WWF.
In scenario B, when I watch the system performance, CPU goes up to 80% for a couple of seconds and then it drops. After that no activity for 30-40s before it actually starts processing each child WWF process.
So where is the bottleneck/slowness and how do we resolve this occasional pause in action?
Any suggestions?
Thanks for your time.