native-lzo library not available on Hadoop datanodes

Question

I've written a simple LzoWordCount the following to my Gateway/hadoop-env.sh:

HADOOP_CLASSPATH=/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
JAVA_LIBRARY_PATH=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.105/lib/hadoop/lib/native/

When I run the MR job, I get:

mapred.JobClient: Task Id : attempt_201307311800_0020_m_000002_2, Status : FAILED java.lang.RuntimeException: native-lzo library not available

Any ideas how to fix this issue? I did notice that 'hadoop classpath | grep native' returns nothing?

Answer 1

goto: https://code.google.com/p/hadoop-gpl-packing/downloads/list

download the jar and the rpm install the rpm (at the time: hadoop-gpl-packaging-0.6.1-1.x86_64.rpm) and then

yum install install lzop
rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm
sudo cp /opt/hadoopgpl/native/Linux-amd64-64/* /usr/lib/hadoop/lib/native/
sudo cp /opt/hadoopgpl/lib/*lzo* /usr/lib/hadoop/lib/

be weary with the above commands not to override files you don't want to. so first make a back of the target folder before you run this.

Answer 2

The problem turned out to be that we did not have lzop installed on the datanodes. I fixed it using:

sudo apt-get install lzop

Answer 3

So it means your hadoop-lzo-***.jar is not in your HADOOP_CLASSPATH

You can then copy your jar to your hadoop/lib folder or add the path to you hadoop/bin/hadoop-env.sh, you have to do it on ALL your nodes

I hope it will solve it ;-)

Answer 4

1.install lzo

yum install lzo -y

2.install hadoop-gpl-packaging

wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/hadoop-gpl-packing/hadoop-gpl-packaging-0.6.1-1.x86_64.rpm
rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm

3.copy jar

cp /opt/hadoopgpl/lib/hadoop-lzo-*.jar $HADOOP_HOME/share/hadoop/common/

4 copy lib

cp /opt/hadoopgpl/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/