Installable Python tar.gz package depends on OS used to build tarball

Question

I recently encountered the odd scenario where, I wanted to convert my (correctly formatted) Python package coolpackage into a tarball for simple distribution to some colleagues, yet the validity of the resulting tarball depends on the OS used to generate it.

For context, coolpackage can be installed from the source directory via pip on either macOS and Ubuntu Linux without issue:

cd coolpackage
pip install .

This works on both macOS and Linux, and in both cases is being run in a conda environment that is identical in terms of packages.

Now, if I generated a tarball in Ubuntu

# run inside Ubuntu 18.04
tar -czvf coolpackage.tar.gz coolpackage

Then that tar.gz file can be installed using pip on both macOS and Ubuntu:

pip install coolpackage.tar.gz

However, if I generate a tarball in macOS

# run inside macOS Mojave
tar -czvf coolpackage.tar.gz coolpackage

That tarball is not installable on either macOS or Ubuntu and exits with basically the same error in both cases (paths notwithstanding):

    pip install coolpackage.tar.gz
    Processing ./coolpackage.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/alex/Python/conda/envs/localconda/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/2n/xtzsyspd32v6vglg_pd5gmw80000gn/T/pip-req-build-tz8wnxlx/setup.py'"'"'; __file__='"'"'/private/var/folders/2n/xtzsyspd32v6vglg_pd5gmw80000gn/T/pip-req-build-tz8wnxlx/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/2n/xtzsyspd32v6vglg_pd5gmw80000gn/T/pip-req-build-tz8wnxlx/pip-egg-info
         cwd: /private/var/folders/2n/xtzsyspd32v6vglg_pd5gmw80000gn/T/pip-req-build-tz8wnxlx/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/alex/Python/conda/envs/localconda/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/2n/xtzsyspd32v6vglg_pd5gmw80000gn/T/pip-req-build-tz8wnxlx/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Wondering if anyone has encountered this before, or has thoughts on what might be going on? I can avoid this by building in Linux, so not an immediate issue, but I don't like not understanding why something fails.

Answer 1

There are various dialects / flavors of tar that are largely similar. In this case we're looking at GNU tools commonly used on Linux systems and BSD tools used on BSD (-> Darwin -> MacOS X).

If you want run BSD tar on Linux try bsdtar and if you want to use GNU tar on BSD (but also some other U*X systems), try gtar.

This also translate into other consumers of tarballs and it seems to be what trips pip which apparently expects GNU version (gtar) flavored inputs.

Answer 2

Since you are using Python, it could be a good idea to directly use python -m tarfile and python -m gzip from its standard library. It should be portable enough.

Also, assuming the project is packaged with setuptools (or distutils) the using the sdist directly should be good enough in most cases (python setup.py sdist).