Cannot copy files from local file system to hadoop/hdfs on cloudera

Question

After installing cloudera HDC on fedora25 , I can create folders, but not files nor can I copy data from my local file system to HDFS.

This is the command I use:

sudo -u hdfs hadoop fs -copyFromLocal /home/mohammed/Documents/bbc.txt /kareem/corpora/

and this is what I get from the terminal:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
copyFromLocal: '/home/mohammed/Documents/bbc.txt': No such file or directory

How to overcome this problem?

Your kind help is highly appreciated!

Answer 1

The problem is that your local path of /home/mohammed is not accessible as the user hdfs you're sudo-ing as to run the whole command. Since the local linux user of hdfs cannot enter /home/mohammed, the command throws a No such file or directory error and exits as a result of being unable to locate or read the provided file.

In most packaged HDFS installations, the hdfs user is typically the super-user of the distributed filesystem and administrative commands are typically run as that user. However, doing work over data can and should be done as regular users after using the hdfs user to provision permissions and ownerships for regular users.

For your case, you can do the following as your mohammed user, if this account also has sudo privileges:

# Superuser-provisioning part (do once)

# Ensure the HDFS directory exists by creating it as a superuser
~> sudo -u hdfs hadoop fs -mkdir -p /kareem/corpora
# Ensure also the HDFS-home path exists by creating it as a superuser
~> sudo -u hdfs hadoop fs -mkdir -p /user/mohammed
# Grant ownership entirely to user mohammed for both paths
~> sudo -u hdfs hadoop fs -chown -R mohammed:mohammed /kareem /user/mohammed

# Final usage part (continue or repeat as many times) without superuser

# Upload the local file (note the absence of sudo)
~> hadoop fs -copyFromLocal -f /home/mohammed/Documents/bbc.txt /kareem/corpora/
# Now read it, etc., all done as the regular non-'hdfs' user
~> hadoop fs -text /home/mohammed/Documents/bbc.txt