Python EGG: include top-level dir

Question

We have something similar to this:

/dir/
    subdir1/
           subdir1-1/
                    (some module)
                    setup.py
           subdir1-2/
                    (another module)
    subdir2/
           (another module)

The code from subdir1-2 uses (=imports) code from subdir2 module. Everything's fine until we want to pack subdir2 into EGG (from subdir1-1) and re-use it. It seems EGG can't pack anything from higher-level dir(s).

Our setup.py:

data_files=[
    (root, [os.path.join(root, f) for f in files])
    for root, _, files in itertools.chain(
        os.walk('some_module'),
        os.walk('../subdir2'),  # DOES NOT WORK!
        os.walk('../../subdir2),  # DOES NOT WORK!
        os.walk(os.path.join(current_dir, '..', 'subdir2')),  # DOES NOT WORK
        os.walk(os.path.join(current_dir, '..', '..', 'subdir2'))  # DOES NOT WORK AS WELL!
    )
],

We also tried to add the module to sys.path - didn't work. Any module that is located in an inner directory works well. Any module from a higher-level directory - does not work.

I'm writing it by memory so please don't look for mistypes in the code I provided above - in our code, we wrote 100% correct code - no typos, no wrong-level dirs (like .. instead of ../..).

The problem is exactly this: higher-level dirs are not included into the EGG; or we can't use them.

The answer from here: top-level package handling with setuptools (or another python egg builder) does not work.

Scrapyd 0.24, Python 2.7 (standard Ubuntu 14.04 x64 package).

Thanks!

P.S. We're using Scrapyd which packs the code into EGGs. And it's just really painful to debug such a "smart" solution (after all, who decided to use such a nightmare'ish deployment solution? If only we could throw away Scrapyd...).

Answer 1

Move setup.py to /dir/ and rewrite all your stuff accordingly.

The way it works subdir1-1 is the root of your build tree so no, you can't go above that. Works inside the resulting egg, too, so you can't include higher-level dirs and you couldn't use them if you did.