Why does querying an external hive table require write access to the hdfs directory?

1k views Asked by At

I've hit an interesting permissions problem when setting up an external table to view some Avro files in Hive.

The Avro files are in this directory :

drwxr-xr-x   - myserver hdfs          0 2017-01-03 16:29 /server/data/avrofiles/

The server can write to this file, but regular users cannot.

As the database admin, I create an external table in Hive referencing this directory:

hive> create external table test_table (data string) stored as avro location '/server/data/avrofiles';

Now as a regular user I try to query the table:

hive> select * from test_table limit 10;
FAILED: HiveException java.security.AccessControlException: Permission denied: user=regular.joe, access=WRITE, inode="/server/data/avrofiles":myserver:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

Weird, I'm only trying to read the contents of the file using hive, I'm not trying to write to it.

Oddly, I don't get the same problem when I partition the table like this:

As database_admin:

hive> create external table test_table_partitioned (data string) partitioned by (value string) stored as avro;
OK
Time taken: 0.104 seconds
hive> alter table test_table_partitioned add if not exists partition (value='myvalue') location '/server/data/avrofiles';
OK

As a regular user:

hive> select * from test_table_partitioned where value = 'some_value' limit 10;
OK

Can anyone explain this?

One interesting thing I noticed is that the Location value for the two tables are different and have different permissions:

hive> describe formatted test_table;
Location:               hdfs://server.companyname.com:8020/server/data/avrofiles

$ hadoop fs -ls /apps/hive/warehouse/my-database/
drwxr-xr-x   - myserver hdfs          0 2017-01-03 16:29 /server/data/avrofiles/

user cannot write

hive> describe formatted test_table_partitioned;
Location:               hdfs://server.companyname.com:8020/apps/hive/warehouse/my-database.db/test_table_partitioned

$ hadoop fs -ls /apps/hive/warehouse/my-database.db/
drwxrwxrwx   - database_admin         hadoop          0 2017-01-04 14:04 /apps/hive/warehouse/my-database.db/test_table_partitioned

anyone can do anything :)

0

There are 0 answers