I am trying to run simple pagerank labtask on my hadoop 3.3.6 installed on ubuntu virtual box but it is giving this error while all my commands are true and my instructor just tole me to download hadoop again because we cant figure out this error Can anyone help?terminal screenshot here these are the command lines
hadoop jar Home/hadoop-3.3.6/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar -input /inputs7/input3.txt -output /inputs7/input3.txt -mapper mapper1.py -reducer reducer1.py -file /home/amina01/Documents/mapper1.py -file /home/amina01/Documents/mapper1.py
These are my commands I started from the beginning now It is giving even more newer ones
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -mkdir -p /user/amina01
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -put -f /home/amina01/Documents/input.txt
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -put -f /home/amina01/Documents/input.txt /user/amina01
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -mkdir -p /input/output
amina01@amina01-VirtualBox:~/Documents$ python3 driver.py /input/input.txt /input/output /home/amina01/Documents/mapper.py /home/amina01/Documents/reducer.py 5
and the error goes like
cp: `/input/input.txt': No such file or directory
2024-03-13 23:55:07,241 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [/home/amina01/Documents/mapper.py, /home/amina01/Documents/reducer.py, /tmp/hadoop-unjar7254882529784803509/] [] /tmp/streamjob4354537991983504292.jar tmpDir=null
2024-03-13 23:55:08,512 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2024-03-13 23:55:08,885 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2024-03-13 23:55:09,195 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/amina01/.staging/job_1710346569620_0001
2024-03-13 23:55:09,934 INFO mapred.FileInputFormat: Total input files to process : 0
2024-03-13 23:55:10,056 INFO mapreduce.JobSubmitter: number of splits:0
2024-03-13 23:55:10,352 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1710346569620_0001
2024-03-13 23:55:10,353 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-03-13 23:55:10,756 INFO conf.Configuration: resource-types.xml not found
2024-03-13 23:55:10,757 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-03-13 23:55:11,160 INFO impl.YarnClientImpl: Submitted application application_1710346569620_0001
2024-03-13 23:55:11,300 INFO mapreduce.Job: The url to track the job: http://amina01-VirtualBox:8088/proxy/application_1710346569620_0001/
2024-03-13 23:55:11,306 INFO mapreduce.Job: Running job: job_1710346569620_0001
2024-03-13 23:55:21,526 INFO mapreduce.Job: Job job_1710346569620_0001 running in uber mode : false
2024-03-13 23:55:21,530 INFO mapreduce.Job: map 0% reduce 0%
2024-03-13 23:55:27,754 INFO mapreduce.Job: Task Id : attempt_1710346569620_0001_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
2024-03-13 23:55:32,828 INFO mapreduce.Job: Task Id : attempt_1710346569620_0001_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
2024-03-13 23:55:38,895 INFO mapreduce.Job: Task Id : attempt_1710346569620_0001_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
2024-03-13 23:55:44,938 INFO mapreduce.Job: map 0% reduce 100%
2024-03-13 23:55:46,023 INFO mapreduce.Job: Job job_1710346569620_0001 failed with state FAILED due to: Task failed task_1710346569620_0001_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0
2024-03-13 23:55:46,200 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=12214
Total time spent by all reduce tasks (ms)=12214
Total vcore-milliseconds taken by all reduce tasks=12214
Total megabyte-milliseconds taken by all reduce tasks=12507136
2024-03-13 23:55:46,200 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!