Pig 0.12.0 won't execute shell commands with timezone change using backticks

389 views Asked by At

I'm using Hue for PIG scripts on amazon EMR. I want to make a shell call to get the date in a particular timezone into a variable which I will use to define an output folder path for writing the output to. Eventually I want to use a if else fi loop to get a particular date from a week, so the time zone will be mentioned at various places in the command.

Sample Script

ts = LOAD 's3://testbucket1/input/testdata-00000.gz' USING PigStorage('\t');
STORE ts INTO 's3://testbucket1/$OUTPUT_FOLDER' USING PigStorage('\t');

Pig parameter definition in Hue:

  1. This works: OUTPUT FOLDER = `/bin/date +%Y%m%d`
  2. This doesn't work: OUTPUT FOLDER = `TZ=America/New_York /bin/date +%Y%m%d`

Both of the commands execute perfectly in the bash shell. But the second command gives the following error:

2015-06-23 21:43:42,901 [main] INFO org.apache.pig.tools.parameters.PreprocessorContext - Executing command : TZ=America/Phoenix /bin/date +%Y%m%d 2015-06-23 21:43:42,913 [main] ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. Error executing shell command: TZ=America/Phoenix /bin/date +%Y%m%d. Command exit with exit code of 126

From the GNU manual: If a command is found but is not executable, the return status is 126.

How do I resolve this?

Configuration details:

  • AMI version:3.7.0
  • Hadoop distribution:Amazon 2.4.0
  • Applications:Hive 0.13.1, Pig 0.12.0, Impala 1.2.4, Hue
  • Underlying shell: bash
  • User: hadoop (while using Pig and while using Bash)

If you need any clarifications then please do comment on this question. I will update it as needed.

EDIT: Under the hood, Pig calculates the value by executing "bash -c exec (command)" and assigning it to the variable, where (command) is whatever we put as a value for the variable in Hue

1

There are 1 answers

8
Romain On

If I do:

date --date='TZ="America/Los_Angeles"' '+%Y%m%d'
20150624

e.g.

%default date_dir `date --date='TZ="America/Los_Angeles"' '+%Y%m%d'`;