TechQA.

What should I do to use python-magic on Heroku?

486 views Asked by At

I'm building an application that is using python-magic 0.4.15. On my local machine, python-magic-bin 0.4.15 works like a charm (Windows) but I'm trying to make it work on Heroku. First I'm just trying to get the tests to pass in the Heroku-CI. I added the package to my requirements.txt:

python-magic==0.4.15

And pushed it to Heroku. And it gets installed correctly:

-----> Installing requirements with pip
       Collecting python-magic==0.4.15 (from -r /app/requirements.txt (line 7))
         Downloading https://files.pythonhosted.org/packages/42/a1/76d30c79992e3750dac6790ce16f056f870d368ba142f83f75f694d93001/python_magic-0.4.15-py2.py3-none-any.whl
       Installing collected packages: python-magic
       Successfully installed python-magic-0.4.15

But then it crashes like this:

+ python manage.py collectstatic --no-input
Traceback (most recent call last):
  File "manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/registry.py", line 120, in populate
    app_config.ready()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/apps.py", line 24, in ready
    self.module.autodiscover()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/__init__.py", line 26, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/core/admin.py", line 8, in <module>
    from .forms import CrmDataUploadForm
  File "/app/core/forms.py", line 3, in <module>
    from magic import magic
ImportError: cannot import name 'magic'

I tried adding the NetsoftHoldings/heroku-buildpack-magic buildpack and I can see it's being downloaded:

-----> Fetching https://github.com/NetsoftHoldings/heroku-buildpack-magic.git buildpack...
       buildpack downloaded

The pip package is being correctly installed:

-----> Installing requirements with pip
       Collecting python-magic==0.4.15 (from -r /app/requirements.txt (line 7))
         Downloading https://files.pythonhosted.org/packages/42/a1/76d30c79992e3750dac6790ce16f056f870d368ba142f83f75f694d93001/python_magic-0.4.15-py2.py3-none-any.whl
       Installing collected packages: python-magic
       Successfully installed python-magic-0.4.15

but then there seems to be an error with the buildpack (I'm guessing here):

-----> libmagic app detected
bash: /tmp/buildpacks/27353aa3b5edb2f8361186206e863f5d481f02805d02f0efdec439f3d30349a4fbdad2951af33a0492023fb0d90d26dafda6fe76f1b2d34f68564a98545f7022/export: No such file or directory
RootDir: /tmp/buildpacks/ba9220b10d71b7b776013869da2c820ccb5c0f1733268ae2515b3b8ad5edd44d2016d1119bce32d1660fbd84da179891a404dcc57df043b9d82e22725739b072
BuildDir: /app
HOME: /app
Build list
config
Home list
config

and it eventually crashes like this:

+ python manage.py collectstatic --no-input
Traceback (most recent call last):
  File "manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/registry.py", line 120, in populate
    app_config.ready()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/apps.py", line 24, in ready
    self.module.autodiscover()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/__init__.py", line 26, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/core/admin.py", line 8, in <module>
    from .forms import CrmDataUploadForm
  File "/app/core/forms.py", line 3, in <module>
    from magic import magic
ImportError: cannot import name 'magic'

Then I tried adding the heroku-community/apt buildpack and an Aptfile containing:

libmagic-dev
libmagic1

and I can see them being installed correctly:

-----> Fetching .debs for libmagic-dev
       Reading package lists...
       Building dependency tree...
       0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 23 not upgraded.
       Need to get 79.4 kB of archives.
       After this operation, 0 B of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 libmagic-dev amd64 1:5.32-2ubuntu0.1 [79.4 kB]
       Fetched 79.4 kB in 0s (222 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
-----> Fetching .debs for libmagic1
       Reading package lists...
       Building dependency tree...
       0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 23 not upgraded.
       Need to get 68.4 kB of archives.
       After this operation, 0 B of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 libmagic1 amd64 1:5.32-2ubuntu0.1 [68.4 kB]
       Fetched 68.4 kB in 0s (199 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
-----> Installing libmagic1_1%3a5.32-2ubuntu0.1_amd64.deb
-----> Installing libmagic-dev_1%3a5.32-2ubuntu0.1_amd64.deb

But the application still fails with the same error:

+ pip install -r requirements-test.txt
Requirement already satisfied: coverage==4.5.1 in ./.heroku/python/lib/python3.6/site-packages (from -r requirements-test.txt (line 1))
Requirement already satisfied: selenium==3.141.0 in ./.heroku/python/lib/python3.6/site-packages (from -r requirements-test.txt (line 2))
Requirement already satisfied: urllib3 in ./.heroku/python/lib/python3.6/site-packages (from selenium==3.141.0->-r requirements-test.txt (line 2))
You are using pip version 9.0.2, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ python manage.py collectstatic --no-input
Traceback (most recent call last):
  File "manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/registry.py", line 120, in populate
    app_config.ready()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/apps.py", line 24, in ready
    self.module.autodiscover()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/admin/__init__.py", line 26, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/core/admin.py", line 8, in <module>
    from .forms import CrmDataUploadForm
  File "/app/core/forms.py", line 3, in <module>
    from magic import magic
ImportError: cannot import name 'magic'

Any ideas how to get python-magic working on Heroku?

python heroku python-magic
Original Q&A
1

There are 1 answers

1
ewwink On BEST ANSWER

The correct way to import the magic module is:

import magic

not

from magic import magic

python-magic just provides a magic module and python-magic-bin does the same. The problem is that python-magic-bin has a package called magic that imports the module magic.magic, which is sort-of like an internal detail, hence, the latter example works in python-magic-bin but not python-magic.

From the python-magic documentation:

>>> import magic
>>> magic.from_file("testdata/test.pdf")
'PDF document, version 1.2'
>>> magic.from_buffer(open("testdata/test.pdf").read(1024))
'PDF document, version 1.2'
>>> magic.from_file("testdata/test.pdf", mime=True)
'application/pdf'

Related Questions in PYTHON

Related Questions in HEROKU

Related Questions in PYTHON-MAGIC

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions