I am fairly new to using hadoop and I've been get these exceptions when I run a file on hadoop.Please help. this is the command: hadoop jar /home/eeman/hadoop-3.2.4/share/hadoop/tools/lib/hadoop-streaming-3.2.4.jar -mapper "python3 /home/hadoop/mapper.py" -reducer "python3 /home/hadoop/reducer.py" -input /user/input.txt -output /output2
I expected that an output file would be created with the result of the word count problem. But, the output directory was empty.
packageJobJar: [/tmp/hadoop-unjar268966945647484645/] [] /tmp/streamjob503043982839100895.jar tmpDir=null 2023-03-05 18:55:39,789 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032 2023-03-05 18:55:39,937 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032 2023-03-05 18:55:40,127 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/eeman/.staging/job_1678024500909_0001 2023-03-05 18:55:40,834 INFO mapred.FileInputFormat: Total input files to process : 1 2023-03-05 18:55:40,887 INFO mapreduce.JobSubmitter: number of splits:2 2023-03-05 18:55:41,011 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1678024500909_0001 2023-03-05 18:55:41,013 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-03-05 18:55:41,157 INFO conf.Configuration: resource-types.xml not found 2023-03-05 18:55:41,157 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-03-05 18:55:41,532 INFO impl.YarnClientImpl: Submitted application application_1678024500909_0001 2023-03-05 18:55:41,612 INFO mapreduce.Job: The url to track the job: http://eeman-ThinkPad-T480:8088/proxy/application_1678024500909_0001/ 2023-03-05 18:55:41,613 INFO mapreduce.Job: Running job: job_1678024500909_0001 2023-03-05 18:55:47,702 INFO mapreduce.Job: Job job_1678024500909_0001 running in uber mode : false 2023-03-05 18:55:47,705 INFO mapreduce.Job: map 0% reduce 0% 2023-03-05 18:55:51,798 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:55:51,828 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:55:54,847 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000001_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:55:54,849 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:55:59,886 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000001_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:56:00,894 INFO mapreduce.Job: Task Id : attempt_1678024500909_0001_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2023-03-05 18:56:02,909 INFO mapreduce.Job: map 100% reduce 100% 2023-03-05 18:56:02,933 INFO mapreduce.Job: Job job_1678024500909_0001 failed with state FAILED due to: Task failed task_1678024500909_0001_m_000001 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2023-03-05 18:56:03,005 INFO mapreduce.Job: Counters: 11 Job Counters Failed map tasks=7 Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=7 Other local map tasks=5 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=15240 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=15240 Total vcore-milliseconds taken by all map tasks=15240 Total megabyte-milliseconds taken by all map tasks=15605760 2023-03-05 18:56:03,005 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!