How to set spark.executor.extraClassPath & spark.driver.extraClassPath in hive query without adding those in hive-site.xml

25 views Asked by At

I Have a hql file which is being used in oozie hive action. I am running hive on spark engine by specifying the below in my hql file.

set hive.execution.engine=spark;

But the job is failing with error : java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree

Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.CommonTree

I researched about it and trying to solve this by setting spark.executor.extraClassPath & spark.driver.extraClassPath

I found this can be done by adding the entry in hive-site.xml . But I am trying to set this on run time by adding the below command in my hql file like below :

set spark.executor.extraClassPath=/abc/external_jars/antlr-runtime-4.7.jar; set spark.driver.extraClassPath=/abc/external_jars/antlr-runtime-4.7.jar;

But how can I add multiple jar file like this or is this the correct way to set the Classpath in runtime via hive ?

Thanks

Tried to run hive query via oozie action by setting the engine as spark. But it is failing with Exception : java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree

Expecting to run the job successfully by correctly identifying the antlr jar file.

Thanks

0

There are 0 answers