What happens to orphaned Yarn Child processes?

Question

Hadoop YARN launches instances of YarnChild in child VM to execute the actual tasks. Those tasks communicate with their ApplicationMaster (AM) through the umbilical interface.

My question is what happens if AM dies and Resource Manager(RM) fails to bring it up (say, due to some code defect in AM)? In such a case, the children tasks would (a) note the absence of AM due to heartbeat and then, (b) go to RM to get new AM location, which in this case they will not get. So, what happens to these orphaned tasks? I have a scenario where I would like to terminate them. Is that the default behavior and does their NodeManager (NM) terminate them?

Answer 1

All child task related to that particular application master would be on halt state. Hadoop admin should either restart the application master or kill it. NodeManager doesn't terminate the failed Application Master.

If you want to kill a application then you can use yarn application -kill application_id command to kill the application. It will kill all running and queued jobs under the application.

If you want to kill a task in YARN then you can use hadoop job -kill-task <task-id> to kill a particular task in YARN

Answer 2

From Hadoop -Definitive Guide, Chapter 6, Failures, Failures in yarn

After a crash, a new resource manager instance is brought up(by admin), and it recovers from the saved state. The state consists of node managers in system, as well as running applications. Here tasks are not part of resource managers state, as they are managed by application.

Also, it is said that the resource manager is designed to be able to recover from crashes.