In distributed jmeter load testing, got stuck at 'Waiting for possible shutdown message on port 4445'

Question

I have added 16 Concurrency thread group in my test plan and when running with 2K users distributed across 19 slave server I am able to see the summary report on the command prompt when run the command tail -f -n 2 MVP_2K_Run.log But when run the same test plan with 15K thread (users) across the 19 slave server and run the same tail command then stuck at "Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445" and there is no summary showing on the command prompt.

I have run my test plan with 15K users and distributed on 19 salve servers using below command

nohup /opt/apache-jmeter-5.4.1/bin/jmeter -Djava.rmi.server.hostname=10.104.180.50 -n -t Novus_MVP5.2_BaseLinePT_15K.jmx -l Novus_MVP5.2_BaseLinePT_15K_23Oct.jtl -R 10.104.180.51:4000,10.104.180.52:4000,10.104.180.53:4000,10.104.180.54:4000,10.104.180.55:4000,10.104.180.56:4000,10.104.180.57:4000,10.104.180.58:4000,10.104.180.59:4000,10.104.180.60:4000,10.104.180.61:4000,10.104.180.62:4000,10.104.180.63:4000,10.104.180.64:4000,10.104.180.65:4000,10.104.180.66:4000,10.104.180.67:4000,10.104.180.68:4000,10.104.180.69:4000 > Novus_MVP5.2_BaseLinePT_15K_23Oct.log 2>&1 &

JTL and log files are got created on server under the specified path but summary is not showing on the command prompt when runs the below command

tail -f -n 2 Novus_MVP5.2_BaseLinePT_15K_23Oct.log

Its showing the below response

Remote engines have been started:[10.104.180.51:4000, 10.104.180.54:4000, 10.104.180.68:4000, 10.104.180.62:4000, 10.104.180.55:4000, 10.104.180.65:4000, 10.104.180.57:4000, 10.104.180.52:4000, 10.104.180.60:4000, 10.104.180.63:4000, 10.104.180.66:4000, 10.104.180.59:4000, 10.104.180.67:4000, 10.104.180.58:4000, 10.104.180.64:4000, 10.104.180.69:4000, 10.104.180.53:4000, 10.104.180.56:4000, 10.104.180.61:4000] Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445

Answer 1

I don't think you can evenly distribute 2000 or 15000 threads between 19 slaves.

If you have 2000 users in Thread Groups and 19 slaves the total load is 38000.

If you have 15000 users in Thread Groups and 19 slaves the total load is 285000.

If you need to execute your test with 285000 concurrent users and it hangs:

1. Check jmeter-server.log files on slaves for any suspicious entries
2. Make sure to follow recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure to properly tune your JMeter instances for high loads
3. Take thread and heap dumps and see where exactly your test is stuck. It might be the case your system under test is overloaded and doesn't respond and JMeter by default waits for the response forever, it might worth defining reasonable timeouts in your Samplers