How to correctly install PyICU on Heroku?

1.1k views Asked by At

I am trying to deploy my Python app on Heroku, but have been unsuccessful. It seems that a problem is occurring with the PyICU package, which I'm unsure how to correct. I've confirmed that this is the only issue with my deployment; when I remove PyICU from my requirements file, everything works. But of course my site can't work without it.

Can anyone please guide me in how to correctly install this package on Heroku? I've tried various methods, including downloading the .whl file and then adding that to my requirements file, but then I get another error:

ERROR: PyICU-2.7.3-cp38-cp38m-win_amd64.whl is not a supported wheel on this platform. I don't understand why - it's the correct Python and os version.

Here are the relevant excerpts from the build log:

-----> Building on the Heroku-20 stack
-----> Using buildpack: heroku/python
-----> Python app detected
-----> Using Python version specified in runtime.txt
-----> Installing python-3.8.10
-----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2
-----> Installing SQLite3
-----> Installing requirements with pip
       ...
         Building wheel for PyICU (setup.py): started
         Building wheel for PyICU (setup.py): finished with status 'error'
         ERROR: Command errored out with exit status 1:
          command: /app/.heroku/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-zeqs7m46
              cwd: /tmp/pip-install-e4xp9bv_/pyicu/
         Complete output (90 lines):
         (running 'icu-config --version')
         (running 'pkg-config --modversion icu-i18n')
         
         Building PyICU 2.4.3 for ICU 66.1
         
         (running 'icu-config --cxxflags --cppflags')
         Could not configure CFLAGS with icu-config
         (running 'pkg-config --cflags icu-i18n')
         (running 'icu-config --ldflags')
         Could not configure LFLAGS with icu-config
         (running 'pkg-config --libs icu-i18n')
         Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
         running bdist_wheel
         running build
         running build_py
         creating build
         creating build/lib.linux-x86_64-3.7
         copying PyICU.py -> build/lib.linux-x86_64-3.7
         creating build/lib.linux-x86_64-3.7/icu
         copying icu/__init__.py -> build/lib.linux-x86_64-3.7/icu
         running build_ext
         building '_icu' extension
         creating build/temp.linux-x86_64-3.7
         gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/app/.heroku/python/include/python3.7m -c _icu.cpp -o build/temp.linux-x86_64-3.7/_icu.o -DPYICU_VER="2.4.3"
         In file included from /usr/include/c++/9/clocale:42,
                          from /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:41,
                          from /usr/include/c++/9/bits/localefwd.h:40,
                          from /usr/include/c++/9/string:43,
                          from /usr/include/unicode/std_string.h:37,
                          from /usr/include/unicode/unistr.h:38,
                          from common.h:106,
                          from _icu.cpp:27:
         ./locale.h:29:23: error: ‘Locale’ was not declared in this scope; did you mean ‘locale_t’?
            29 | PyObject *wrap_Locale(Locale *locale, int flags);
               |                       ^~~~~~
               |                       locale_t
         ./locale.h:29:31: error: ‘locale’ was not declared in this scope; did you mean ‘locale_t’?
            29 | PyObject *wrap_Locale(Locale *locale, int flags);
               |                               ^~~~~~
               |                               locale_t
         ./locale.h:29:39: error: expected primary-expression before ‘int’
            29 | PyObject *wrap_Locale(Locale *locale, int flags);
               |                                       ^~~
         ./locale.h:29:48: error: expression list treated as compound expression in initializer [-fpermissive]
            29 | PyObject *wrap_Locale(Locale *locale, int flags);
               |                                                ^
         ./locale.h:30:29: error: ‘Locale’ does not name a type; did you mean ‘locale_t’?
            30 | PyObject *wrap_Locale(const Locale &locale);
               |                             ^~~~~~
               |                             locale_t
         ./locale.h:30:43: error: ‘PyObject* wrap_Locale(const int&)’ redeclared as different kind of entity
            30 | PyObject *wrap_Locale(const Locale &locale);
               |                                           ^
         ./locale.h:29:11: note: previous declaration ‘PyObject* wrap_Locale’
            29 | PyObject *wrap_Locale(Locale *locale, int flags);
               |           ^~~~~~~~~~~
         In file included from /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:41,
                          from /usr/include/c++/9/bits/localefwd.h:40,
                          from /usr/include/c++/9/string:43,
                          from /usr/include/unicode/std_string.h:37,
                          from /usr/include/unicode/unistr.h:38,
                          from common.h:106,
                          from _icu.cpp:27:
         /usr/include/c++/9/clocale:53:11: error: ‘::lconv’ has not been declared
            53 |   using ::lconv;
               |           ^~~~~
         /usr/include/c++/9/clocale:54:11: error: ‘::setlocale’ has not been declared
            54 |   using ::setlocale;
               |           ^~~~~~~~~
         /usr/include/c++/9/clocale:55:11: error: ‘::localeconv’ has not been declared
            55 |   using ::localeconv;
               |           ^~~~~~~~~~
         In file included from /usr/include/c++/9/bits/localefwd.h:40,
                          from /usr/include/c++/9/string:43,
                          from /usr/include/unicode/std_string.h:37,
                          from /usr/include/unicode/unistr.h:38,
                          from common.h:106,
                          from _icu.cpp:27:
         /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:52:23: error: ‘uselocale’ was not declared in this scope; did you mean ‘u_fsetlocale’?
            52 |   extern "C" __typeof(uselocale) __uselocale;
               |                       ^~~~~~~~~
               |                       u_fsetlocale
         /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h: In function ‘int std::__convert_from_v(__locale_struct* const&, char*, int, const char*, ...)’:
         /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:75:53: error: ‘__gnu_cxx::__uselocale’ cannot be used as a function
            75 |     __c_locale __old = __gnu_cxx::__uselocale(__cloc);
               |                                                     ^
         /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:100:33: error: ‘__gnu_cxx::__uselocale’ cannot be used as a function
           100 |     __gnu_cxx::__uselocale(__old);
               |                                 ^
         error: command 'gcc' failed with exit status 1
         ----------------------------------------
         ERROR: Failed building wheel for PyICU
         Running setup.py clean for PyICU
       
           Running setup.py install for PyICU: started
           Running setup.py install for PyICU: finished with status 'error'
           ERROR: Command errored out with exit status 1:
            command: /app/.heroku/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bdqgdfr6/install-record.txt --single-version-externally-managed --compile --install-headers /app/.heroku/python/include/python3.7m/PyICU
                cwd: /tmp/pip-install-e4xp9bv_/pyicu/
           Complete output (90 lines):
           (running 'icu-config --version')
           (running 'pkg-config --modversion icu-i18n')
           
           Building PyICU 2.4.3 for ICU 66.1
           
           (running 'icu-config --cxxflags --cppflags')
           Could not configure CFLAGS with icu-config
           (running 'pkg-config --cflags icu-i18n')
           (running 'icu-config --ldflags')
           Could not configure LFLAGS with icu-config
           (running 'pkg-config --libs icu-i18n')
           Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
           running install
           running build
           running build_py
           creating build
           creating build/lib.linux-x86_64-3.7
           copying PyICU.py -> build/lib.linux-x86_64-3.7
           creating build/lib.linux-x86_64-3.7/icu
           copying icu/__init__.py -> build/lib.linux-x86_64-3.7/icu
           running build_ext
           building '_icu' extension
           creating build/temp.linux-x86_64-3.7
           gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/app/.heroku/python/include/python3.7m -c _icu.cpp -o build/temp.linux-x86_64-3.7/_icu.o -DPYICU_VER="2.4.3"
           In file included from /usr/include/c++/9/clocale:42,
                            from /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:41,
                            from /usr/include/c++/9/bits/localefwd.h:40,
                            from /usr/include/c++/9/string:43,
                            from /usr/include/unicode/std_string.h:37,
                            from /usr/include/unicode/unistr.h:38,
                            from common.h:106,
                            from _icu.cpp:27:
           ./locale.h:29:23: error: ‘Locale’ was not declared in this scope; did you mean ‘locale_t’?
              29 | PyObject *wrap_Locale(Locale *locale, int flags);
                 |                       ^~~~~~
                 |                       locale_t
           ./locale.h:29:31: error: ‘locale’ was not declared in this scope; did you mean ‘locale_t’?
              29 | PyObject *wrap_Locale(Locale *locale, int flags);
                 |                               ^~~~~~
                 |                               locale_t
           ./locale.h:29:39: error: expected primary-expression before ‘int’
              29 | PyObject *wrap_Locale(Locale *locale, int flags);
                 |                                       ^~~
           ./locale.h:29:48: error: expression list treated as compound expression in initializer [-fpermissive]
              29 | PyObject *wrap_Locale(Locale *locale, int flags);
                 |                                                ^
           ./locale.h:30:29: error: ‘Locale’ does not name a type; did you mean ‘locale_t’?
              30 | PyObject *wrap_Locale(const Locale &locale);
                 |                             ^~~~~~
                 |                             locale_t
           ./locale.h:30:43: error: ‘PyObject* wrap_Locale(const int&)’ redeclared as different kind of entity
              30 | PyObject *wrap_Locale(const Locale &locale);
                 |                                           ^
           ./locale.h:29:11: note: previous declaration ‘PyObject* wrap_Locale’
              29 | PyObject *wrap_Locale(Locale *locale, int flags);
                 |           ^~~~~~~~~~~
           In file included from /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:41,
                            from /usr/include/c++/9/bits/localefwd.h:40,
                            from /usr/include/c++/9/string:43,
                            from /usr/include/unicode/std_string.h:37,
                            from /usr/include/unicode/unistr.h:38,
                            from common.h:106,
                            from _icu.cpp:27:
           /usr/include/c++/9/clocale:53:11: error: ‘::lconv’ has not been declared
              53 |   using ::lconv;
                 |           ^~~~~
           /usr/include/c++/9/clocale:54:11: error: ‘::setlocale’ has not been declared
              54 |   using ::setlocale;
                 |           ^~~~~~~~~
           /usr/include/c++/9/clocale:55:11: error: ‘::localeconv’ has not been declared
              55 |   using ::localeconv;
                 |           ^~~~~~~~~~
           In file included from /usr/include/c++/9/bits/localefwd.h:40,
                            from /usr/include/c++/9/string:43,
                            from /usr/include/unicode/std_string.h:37,
                            from /usr/include/unicode/unistr.h:38,
                            from common.h:106,
                            from _icu.cpp:27:
           /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:52:23: error: ‘uselocale’ was not declared in this scope; did you mean ‘u_fsetlocale’?
              52 |   extern "C" __typeof(uselocale) __uselocale;
                 |                       ^~~~~~~~~
                 |                       u_fsetlocale
           /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h: In function ‘int std::__convert_from_v(__locale_struct* const&, char*, int, const char*, ...)’:
           /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:75:53: error: ‘__gnu_cxx::__uselocale’ cannot be used as a function
              75 |     __c_locale __old = __gnu_cxx::__uselocale(__cloc);
                 |                                                     ^
           /usr/include/x86_64-linux-gnu/c++/9/bits/c++locale.h:100:33: error: ‘__gnu_cxx::__uselocale’ cannot be used as a function
             100 |     __gnu_cxx::__uselocale(__old);
                 |                                 ^
           error: command 'gcc' failed with exit status 1
           ----------------------------------------
       ERROR: Command errored out with exit status 1: /app/.heroku/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-e4xp9bv_/pyicu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bdqgdfr6/install-record.txt --single-version-externally-managed --compile --install-headers /app/.heroku/python/include/python3.7m/PyICU Check the logs for full command output.
 !     Push rejected, failed to compile Python app.
 !     Push failed

I have a requirements.txt file specifying all the dependencies:

asgiref==3.2.7
astroid==2.3.3
beautifulsoup4==4.9.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
colorama==0.4.3
confusable-homoglyphs==3.2.0
DAWG==0.8.0
DAWG-Python==0.7.2
defusedxml==0.6.0
diff-match-patch==20181111
dj-database-url==0.5.0
Django==3.0.3
django-autoslug-iplweb==1.9.4.dev0
django-bootstrap-form==3.4
django-bootstrap3==12.1.0
django-bootstrap4==1.1.1
django-chartit==0.2.9
django-cleanup==4.0.0
django-countries==6.1.2
django-filter==2.3.0
django-heroku==0.3.1
django-import-export==2.2.0
django-language-field==0.0.3
django-nested-admin==3.3.2
django-registration==3.1
django-restframework==0.0.1
django-tables2==2.3.1
djangorestframework==3.11.0
docopt==0.6.2
et-xmlfile==1.0.1
futures==3.1.1
gunicorn==20.1.0
idna==2.9
isort==4.3.21
jdcal==1.4.1
joblib==0.15.1
lazy-object-proxy==1.4.3
MarkupPy==1.14
mccabe==0.6.1
Morfessor==2.0.6
nltk==3.5
numpy==1.19.0
odfpy==1.4.1
openpyxl==3.0.3
Pillow==7.1.2
pkgconfig==1.5.1
polyglot==16.7.4
progressbar2==3.51.3
psycopg2==2.8.6
pycld2==0.41
# ./PyICU-2.4.3-cp38-cp38-win_amd64.whl   <--I've tried various combinations here
PyICU==2.4.3
pylint==2.4.4
pymorphy2==0.8
pymorphy2-dicts==2.4.393442.3710985
pymystem3==0.2.0
python-dateutil==2.8.1
python-decouple==3.4
python-Levenshtein==0.12.0
python-monkey-business==1.0.0
python-utils==2.4.0
pytz==2019.3
PyYAML==5.3.1
regex==2020.6.8
requests==2.24.0
rutermextract==0.3
six==1.13.0
soupsieve==2.0
sqlparse==0.3.1
tablib==2.0.0
tqdm==4.46.1
unicodecsv==0.14.1
Unidecode==1.1.1
urllib3==1.25.9
whitenoise==5.2.0
wincertstore==0.2
wrapt==1.11.2
xlrd==1.2.0
xlwt==1.3.0

My Procfile looks as follows:

web: gunicorn mysite.wsgi

In runtime.txt I have:

python-3.8.10

I would be very grateful if someone can recommend a fix, thanks. I know that PyICU is not the easiest to install even locally, but how do I go about telling Heroku how to do it?

Edit: I tried adding a custom buildpack for ICU: https://github.com/generalassembly/heroku-icu-buildpack

It seems to run a bit differently then, but still get errors with regard to locale and also the following:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Building ICU 57.1
/tmp/codon/tmp/buildpacks/ad1815f7543324e4f0c006136b48e4be140b5627/bin/compile: line 24: pushd: icu/source: No such file or directory
/tmp/codon/tmp/buildpacks/ad1815f7543324e4f0c006136b48e4be140b5627/bin/compile: line 26: ./runConfigureICU: No such file or directory
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.
/tmp/codon/tmp/buildpacks/ad1815f7543324e4f0c006136b48e4be140b5627/bin/compile: line 30: popd: directory stack empty
Caching build
cp: cannot stat '/app/icu': No such file or directory
2

There are 2 answers

0
person_v1.32 On BEST ANSWER

I haven't tried this myself, but it may be possible to use heroku-buildpack-apt to install PyICU using their Debian installation method.

Otherwise, your wheel needs to be the same version as the server's operating system, not your development machine's OS. You can try building the wheel inside the Heroku dyno using a technique in this StackOverflow answer.

I don't have a repo to test these methods, so I'm not sure how well they'll work. I will be happy to update this answer if you try them out.

0
Alexander Volkovsky On

Why are you using the windows wheel (PyICU-2.7.3-cp38-cp38m-win_amd64.whl)? You probably need a manylinux wheel.

You can also try pyicu-binary package.