Tracing down missing import from py4j on error "trying to call a package"

Question

Given the following python snippet:

from pyspark.mydb.mydb import *
class HBaseTest:

  def __init__(self):
    from pyspark.context import SparkContext
    sc = SparkContext('local[4]', 'PythonTest')
    self._hbctx = MyDbSQLContext(sc)

The line of interest is the last one: the MyDbSQLContext is a class in the org.apache.spark.mydb package. There is a python wrapper for it in the pyspark.mydb.mydb.py file

In an attempt to make the imports known to the py4j java gateway, the following statements are executed:

java_import(self._sc._gateway.jvm,'org.apache.spark.sql.mydb.*')
java_import(self._sc._gateway.jvm,'org.apache.hadoop.conf.Configuration')

However on the last line of the original snippet:

    self._hbctx = MyDbSQLContext(sc)

We get the following error:

sc=org.apache.spark.SparkContext@4925069a conf=<py4j.java_gateway.JavaMember object at 0x10bdd5090>
Traceback (most recent call last):
  File "pyspark/hbase/hbase_tests.py", line 81, in <module>
    test._test()
  File "pyspark/hbase/hbase_tests.py", line 76, in _test
    self.create_test_tables()
  File "pyspark/hbase/hbase_tests.py", line 46, in create_test_tables
    self._ctx().sql(create_sql).toRdd().collect()
  File "/shared/pyjava/python/pyspark/sql.py", line 1620, in sql
    return SchemaRDD(self._ssql_ctx.sql(sqlQuery).toJavaSchemaRDD(), self)
  File "/shared/pyjava/python/pyspark/hbase/hbase.py", line 58, in _ssql_ctx
    self._scala_HBaseSQLContext = self._get_hbase_ctx()
  File "/shared/pyjava/python/pyspark/hbase/hbase.py", line 83, in _get_hbase_ctx
    return self._jvm.HBaseSQLContext(self._jsc.sc())
  File "/shared/pyjava/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 726, in __getattr__
py4j.protocol.Py4JError: Trying to call a package.

UPDATE I have spent more time analyzing the way that imports are performed for hive. They are added via the spark-submit script that internally invokes compute-classpath.sh. I had already correctly added the $MYDB_HOME/lib/* to the classpath in that script. The mydb jar files are available on the classpath:

In addition, the java_imports are being printed - and here they do show the mydb imports:

java_imports: [u'org.apache.hadoop.conf.Configuration', u'org.apache.spark.SparkConf',
u'org.apache.spark.sql.hive.LocalHiveContext', u'org.apache.spark.sql.hive.TestHiveContext', 
u'org.apache.hadoop.mydb.client.Client', u'scala.Tuple2', u'org.apache.spark.sql.mydb.mydbSQLContext', 
u'org.apache.spark.sql.hive.HiveContext', u'java.lang', u'org.apache.hadoop.mydb', u'org.apache.hadoop.mydb.client', 
u'org.apache.hadoop.mydb.filter', u'org.apache.hadoop.mydb.util', u'org.apache.log4j', u'org.apache.spark.api.java', 
u'org.apache.spark.api.python', u'org.apache.spark.mllib.api.python', u'org.apache.spark.sql', u'org.apache.spark.sql.mydb']

It is presently unclear what the source of this issue is. We have verified:

• mydb classes are in the path
• mydb classes are being imported by java_import

• AFAICT the patterns followed for importing and using hive classes are being similarly followed for mydb

  I am at a loss what else to examine to track this java class/classpath issue down.