how to suppress info messages for hadoop fs command

Question

When we run hadoop fs -get command we see all the info messages. How can I suppress these messages.

17/12/05 17:59:02 INFO s3n.S3NativeFileSystem: Opening 's3://testbucketzs/Manish/test1/data/csv-serde-1.1.2-0.11.0-all - Copy (796) - 

Answer 1

You need to modify the file as below :

$HADOOP_HOME/conf/log4j.properties In this file, replace the following line:

log4j.appender.EventCounter=org.apache.hadoop.log.EventCounter

with this line: log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

If you don't want to do that globally point the stderr to the black hole something like this hdfs dfs -get /hdfs_path/ /local_path/ > 2>/dev/null

Another suggestion use hdfs dfs -get instead of hadoop fs its deprecated.

This link has the explanation if you are interested.

Answer 2

If you don't want to globally change the log4j.properties for all the Hadoop processes, you can do a one-off command for

export HADOOP_ROOT_LOGGER="WARN,DFRA"

Then, following hdfs commands in that session will not output INFO messages.