Hadoop with mongoDB : NoClassDefFoundError MongoConfigUtil

Question

I'm learning how to write a map / reduce job in hadoop with mongodb data as input. So I followed this example, but I got following error :

 Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/util/MongoConfigUtil
    at WordCount.main(WordCount.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.util.MongoConfigUtil
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

I've been searching for hours without any result. Any one can help me please.

Answer 1

it means mongo-hadoop-core.jar is available at compile time but not at run time.

Try running your application with dependency jars added in classpath

Example : java -cp mongo-hadoop-core.jar<++other dependencies> MainClass

EDIT 1

If you're running using hadoop shell

check the classpath by executing hadoop classpath it will print the dir/jars in the classpath.

If the dependent jars are not in the classpath add them in the classpath using export command then execute hadoop jar yourjar.jar mainClass

EDIT 2

make use of libjars option

hadoop jar myjar.jar mainClass -libjars ${LIBJARS}

Answer 2

I can see this link examples folder structure is maven. I CANT see pom.xml in that link.

We can set maven scope from provided to runtime

1. You need to write assembly.xml (to package your application lib and related dependencies in tar file) and need to refer it from pom.xml to package mongo-hadoop-core.jar along with other dependencies which are NOT installed in cluster.

export HADOOP_CLASSPATH=`hadoop classpath`:`hbase classpath`

hadoop jar .... -cp $HADOOP_CLASSPATH MainClass

If you unzip the tar file mentioned above and you can refer the classpath for example : hadoop jar .... ../lib/* main classs where lib folder contails all your dependencies like your mongodb also.

2. If mongo db and related components are installed in the cluster we can mention the classpath like below example.

Also see this answer how they have used libjars