MRJob - Limit Number of Task Attemps

Question

In MyJob, how do you limit the number of task attempts (if a task fails)?

I have long running tasks (have increased the timeout, accordingly), but I want the job to end after 2 failed attempts at the same task, rather than 4-5.

I couldn't find anything like this in the docs:

http://mrjob.readthedocs.org/en/latest//en/latest/guides/configs-reference.html

Answer 1

For map jobs, you can set mapreduce.map.maxattempts in Hadoop 2. For reduce jobs, set mapreduce.reduce.maxattempts (source). The equivalents in Hadoop 1 are: mapred.map.max.attempts and mapred.reduce.max.attempts.

If you are using a conf file in MRJob, you can set this as:

runners:
    emr:
        jobconf:
            mapreduce.map.maxattempts: 2