Cannot create temp dir with proper permission: /mnt1/s3

Question

The following is the log dump of one of the container. I got an exception stating that a folder can't be created due to some permissions. I have troubleshooted various time but still it exist.

16/12/19 09:44:05 WARN ConfigurationUtils: Cannot create temp dir with proper permission: /mnt1/s3

java.nio.file.AccessDeniedException: /mnt1 at sun.nio.fs.UnixException.t here ranslateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at com.amazon.ws.emr.hadoop.fs.util.ConfigurationUtils.getTestedTempPaths(ConfigurationUtils.java:224) at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.java:449) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:111) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2717) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2751) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2733) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:377) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:230) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:201)

Answer 1

you should use the same instance type for your master and task/core nodes

see: https://forums.aws.amazon.com/thread.jspa?threadID=57967

Answer 2

I figured it out for my environment. The master node on emr may have the /mnt1 and /mnt1/s3 directory, but ssh into the worker nodes and they're not configured with the proper directories.

I had to change my setup script from a --Step to a --bootstrap-action in the cluster launch command. This allowed me to pass edits onto all worker nodes, even if they scaled up after the initial cluster launch.

Answer 3

I had a similar error when I had an AWS EMR cluster running, and was trying to connect to it from an EMR edge node using RStudio and SparklyR by running a simple "select * from abc" query.

The query worked on the master nodes but not on the edge node. So, I looked at the permissions of /mnt/s3 on the EMR Cluster's master nodes, and compared it with the permissions of that folder in the edge node. The difference was that on the master nodes, the permissions were rwxrwxrwt (like that of /tmp), while on the edge node it was rwxrwxr--.

When I gave the edge node the same expansive permissions as on the master node, the problem disappeared (I also set the sticky bit so that it is exactly the same as on the master node). The commands I used were.

sudo chmod 777 /mnt/s3
sudo chmod o+t /mnt/s3