Run giraph on Hadoop yarn 2.6.0

681 views Asked by At

I'm trying to use Giraph on hadoop 2.6.0 with yarn.

I've managed to build it by removing STATIC_SASL_SYMBOL in <munge.symbols> in the yarn profile. with the command :

sudo mvn -Phadoop_yarn -Dhadoop.version=2.6.0 -DskipTests package

Then i've setup a single node cluster with hadoop 2.6.0 and I'm able to perform the basic wordcount example. I've copied the jar built into the directory share/hadoop/mapreduce to remove the GiraphMaster not found execption.

Now I'm trying to run the SimpleShortestPaths example with the command :

$HADOOP_PREFIX/bin/hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip input_tiny2.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths445 -w 1 -ca giraph.SplitMasterWorker=false -yj giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar

The data used is the tiny graph on the giraph quick start page and the job should takes minutes to complete but it doesn't.

The full log is at http://pastebin.com/CW1FnWz2

It's possible to make giraph work on hadoop 2.6.0 or do I have to use an earlier version ? If so which one would you suggest ?

0

There are 0 answers