Using dill to pickle a class instance and loading in a different package

1k views Asked by At

I am trying to use dill to share objects between two different environments (machine learning research and production).

A simplified example:

package_a/module_a.py:

class P:
  def __init__(self, a, b):
    self.a = a #for example a dict
    self.b = b #a function

def save_A(*, path,a,b):
  import dill
  p = A(a,b)

  dill.dump(p, open(path, "wb"))

package_a/module_b.py:

a = [1,2,3]
def b():
  return 5
from module_a import save_A
save_A("test.p", a , b)

Now I am trying to load this object in a different package in a different repo package_b/prod.py:

import dill
with open("test.p",'rb') as fp:
a = dill.load(fp)

The error I am getting is:

  return StockUnpickler.find_class(self, module, name)
   ImportError: No module named 'module_a'

Dills seems to try and re-import the imports from the module where the save function is called. If I were to include the content from module_b in module_a (for example in a if __name__=="__main__" block) everything works fine.This is not feasible though, because there are a lot of objects that need to get pickled. I am using python 3.5. I don't really understand what might be causing this and why dill remembers the import paths. I could understand if i get an error complaining about A not being in the namespace (even though dill is supposed to package the class definition with the instance) but this I cannot really make sense of.

0

There are 0 answers