I'm trying to dynamically load modules and packages from arbitrary folder locations in python 2.7. It works great with bare, single file modules. But trying to load in a package is a bit harder.
The best I could figure out was to load the init.py file inside the package (folder). But say for example I have this:
root:
mod.py
package:
__init__.py
sub.py
If mod.py contains:
from package import sub
Using my current loading code (below), it will fail stating that there is no package named "sub", unless I add the following to package/__init__.py
import sub
I have to imagine that this is because when you import a package it would normally also scan for all the other sub files in it. Do I also just need to do that manually, or is there a method similar to imp.load_source that will also handle package folders?
Loading code:
import md5
import sys
import os.path
import imp
import traceback
import glob
def load_package(path, base):
try:
try:
sys.path.append(path + "/" + base)
init = path + "/" + base + "/__init__.py"
if not os.path.exists(init):
return None
fin = open(init, 'rb')
return (base, imp.load_source(base, init, fin))
finally:
try: fin.close()
except: pass
except ImportError, x:
traceback.print_exc(file = sys.stderr)
raise
except:
traceback.print_exc(file = sys.stderr)
raise
def load_module(path):
try:
try:
code_dir = os.path.dirname(path)
code_file = os.path.basename(path)
base = code_file.replace(".py", "")
fin = open(path, 'rb')
hash = md5.new(path).hexdigest() + "_" + code_file
return (base, imp.load_source(base, path, fin))
finally:
try: fin.close()
except: pass
except ImportError, x:
traceback.print_exc(file = sys.stderr)
raise
except:
traceback.print_exc(file = sys.stderr)
raise
def load_folder(dir):
sys.path.append(dir)
mods = {}
for p in glob.glob(dir + "/*/"):
base = p.replace("\\", "").replace("/", "")
base = base.replace(dir.replace("\\", "").replace("/", ""), "")
package = load_package(dir, base)
if package:
hash, pack = package
mods[hash] = pack
for m in glob.glob(dir + "/*.py"):
hash, mod = load_module(m)
mods[hash] = mod
return mods
The code below is functionally equivalent to your code modulo the
traceback.print_exc
(which you should let the client handle - if not handled the exception will end up printed anyway):Prints:
mod.py
,sub.py
and__init__.py
just containNow modifying
mod.py
to:we get indeed:
Note the error is "cannot import name sub" and not "there is no package named "sub"". So why can't it ?
Modifying
__init__.py
:prints:
While directly importing it would print:
So modify _load_package to:
Solves it as would:
or in your original code:
So what's going on is that package relies on its
__path __
attribute to function correctly.Kept hacking on that and came up with:
I merged package and modules loading code dropping
imp.load_source
(one less tricky function) and relying on imp.load_module instead. I do not mess with sys.path directly and sinceimp.load_module
will reload [!] I check thesys.modules
cache. Themods
dict returned is completelly untested - you have to somehow implement a hash (the _abspath should suffice).Run as:
to test various scenarios -
The code with an example
root/
dir is here