Does _memimporter still exist in py2exe?

Question

I have only recently started with py2exe, and I would like to use py2exe with MINGW64 Python3 programs (and corresponding libraries). However, the first example I tried failed to build.

After that, I found the https://www.py2exe.org/index.cgi/TroubleshootingImportErrors page, where it is noted:

Check that zipextimporter works on your system

zipextimporter is the starting component of py2exe that may cause problems. To debug it you will need _memimporter.pyd binary module. These modules can be found in binary py2exe distributions for your Python version (I unpack .exe distribution with 7Zip).

There is also a test script, but it is in Python 2 (last update of that page is 2011-01-07); so I converted it to Python 3 syntax:

import zipextimporter
zipextimporter.install()

import sys
sys.path.insert(0, "lib.zip")

import _socket
print(_socket)
# <module '_socket' from 'lib.zip\_socket.pyd'>
print(_socket.__file__)
# 'lib.zip\\_socket.pyd'
print(_socket.__loader__)
# <ZipExtensionImporter object 'lib.zip'>
# Reloading also works correctly:
print(_socket is reload(_socket))
# True

But when I run it, I get:

$ python3 test_zipextimporter.py
Traceback (most recent call last):
  File "D:/msys64/tmp/test_zipextimporter.py", line 1, in <module>
    import zipextimporter
  File "D:/msys64/mingw64/lib/python3.11/site-packages/zipextimporter.py", line 51, in <module>
    import _memimporter
ModuleNotFoundError: No module named '_memimporter'

So, as the note above says, I need a _memimporter.pyd binary module; there is no such file in the package directly:

$ pacman -Ql mingw-w64-x86_64-python-py2exe | grep _mem
$

However, the note also states "These modules can be found in binary py2exe distributions for your Python version (I unpack .exe distribution with 7Zip)." Currently there are these binary files in the package:

$ pacman -Ql mingw-w64-x86_64-python-py2exe | grep '\.exe\|\.dll'
mingw-w64-x86_64-python-py2exe /mingw64/lib/python3.11/site-packages/py2exe/resources.dll
mingw-w64-x86_64-python-py2exe /mingw64/lib/python3.11/site-packages/py2exe/run-py311-mingw_x86_64.exe
mingw-w64-x86_64-python-py2exe /mingw64/lib/python3.11/site-packages/py2exe/run_ctypes_dll-py311-mingw_x86_64.dll
mingw-w64-x86_64-python-py2exe /mingw64/lib/python3.11/site-packages/py2exe/run_w-py311-mingw_x86_64.exe

None of these can be listed with say unzip -l; but they can be listed with 7z l - unfortunately, I see nothing there resembling _memimporter.pyd - both .exes show this structure of files:

$ 7z l /mingw64/lib/python3.11/site-packages/py2exe/run_w-py311-mingw_x86_64.exe
...
   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2023-12-27 10:55:20 .....        24064        24064  .text
2023-12-27 10:55:20 .....         1024         1024  .data
2023-12-27 10:55:20 .....         6656         6656  .rdata
2023-12-27 10:55:20 .....         2048         2048  .pdata
2023-12-27 10:55:20 .....         2048         2048  .xdata
2023-12-27 10:55:20 .....            0            0  .bss
2023-12-27 10:55:20 .....         2048         2048  .edata
2023-12-27 10:55:20 .....         4096         4096  .idata
2023-12-27 10:55:20 .....          512          512  .CRT
2023-12-27 10:55:20 .....          512          512  .tls
                    .....          766          744  .rsrc/1033/ICON/1.ico
                    .....           20           20  .rsrc/1033/GROUP_ICON/1
                    .....         1167         1167  .rsrc/0/MANIFEST/1
2023-12-27 10:55:20 .....          512          512  .reloc
------------------- ----- ------------ ------------  ------------------------
2023-12-27 10:55:20              45473        45451  14 files

I've tried to compare with the vanilla py2exe download, and results are similar:

$ wget https://files.pythonhosted.org/packages/b1/07/f45b201eb8c3fea1af6a9bd9f733479aa9d009139ce2396e06db7aa778c8/py2exe-0.13.0.1-cp311-cp311-win_amd64.whl
# ...
$ mkdir py2exe_0.13.0.1
$ (cd py2exe_0.13.0.1; unzip ../py2exe-0.13.0.1-cp311-cp311-win_amd64.whl)
# ...
$ 7z l py2exe_0.13.0.1/py2exe/run-py3.11-win-amd64.exe
# ...
   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2023-10-07 18:15:11 .....        20480        20480  .text
2023-10-07 18:15:11 .....        11264        11264  .rdata
2023-10-07 18:15:11 .....          512          512  .data
2023-10-07 18:15:11 .....         2048         2048  .pdata
                    .....          766          744  .rsrc/ICON/1.ico
                    .....           20           20  .rsrc/GROUP_ICON/1
                    .....          381          381  .rsrc/MANIFEST/1
2023-10-07 18:15:11 .....          512          512  .reloc
------------------- ----- ------------ ------------  ------------------------
2023-10-07 18:15:11              35983        35961  8 files

I guess this _memimporter is still a thing, because after all my test script fails with "No module named '_memimporter'"; and also there are still references in the Python code of the package:

$ grep -rI _memimporter /mingw64/lib/python3.11/site-packages/py2exe
/mingw64/lib/python3.11/site-packages/py2exe/distutils_buildexe.py:##             self.excludes.append("_memimporter") # builtin in run_*.exe and run_*.dll
/mingw64/lib/python3.11/site-packages/py2exe/hooks.py:# _memimporter can be excluded because it is built into the run-stub.
/mingw64/lib/python3.11/site-packages/py2exe/hooks.py:_memimporter

... but, I have to ask - is this _memimporter.pyd still a thing - and if so, where do I find it?

Answer 1

Right, this was a pain... Anyways, as for the first part of the question - _memimporter is apparently still a thing, since there are recent discussions:

• Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders · Issue #89470 · python/cpython - which caused patch:
• _memimporter: switch to sys.modules dict to check for reimports · py2exe/py2exe@2032a6e

As for getting the _memimporter, it's again tricky, because you have to build from source. There is a historical version https://github.com/mitre/caldera-py2exe - but because of that recent patch above, it's best to build directly from the main py2exe git repo directly:

$ git clone https://github.com/py2exe/py2exe py2exe_git

Now, the question is which command to use to build - unfortunately, the main repo README.md does not specify this (it just mentions pip install py2exe). In the historical repo, a command python setup.py bdist is mentioned, however it seems to be deprecatet, as it raises:

D:/msys64/mingw64/lib/python3.11/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.ins
taller and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  dist.fetch_build_eggs(dist.setup_requires)

I finally found the command that the MINGW64 package uses to build in the corresponding MINGW-packages/mingw-w64-python-py2exe/PKGBUILD file.

Also, note that I had to install:

pacman -S mingw-w64-x86_64-python-build
pacman -S mingw-w64-x86_64-python-wheel

Then, the direct py2exe source, as is, will not compile, so one also has to apply the patches mentioned in the PKGBUILD file. So we have:

$ cd py2exe_git
$ wget https://raw.githubusercontent.com/msys2/MINGW-packages/master/mingw-w64-python-py2exe/001-setup-fix.patch
# ...
$ wget https://raw.githubusercontent.com/msys2/MINGW-packages/master/mingw-w64-python-py2exe/002-mingw-fix.patch
# ...
$ patch -p1 -i 001-setup-fix.patch
patching file py2exe_setuptools.py
Hunk #5 succeeded at 194 with fuzz 1.
patching file setup.py
Hunk #1 succeeded at 23 (offset 2 lines).
Hunk #2 succeeded at 46 (offset 1 line).
Hunk #3 succeeded at 73 (offset 1 line).
Hunk #4 succeeded at 91 (offset 1 line).
Hunk #5 succeeded at 109 (offset 1 line).
Hunk #6 succeeded at 125 (offset 1 line).
$ patch -p1 -i 002-mingw-fix.patch
patching file py2exe/hooks.py
patching file py2exe/runtime.py
Hunk #1 succeeded at 307 (offset 5 lines).
patching file source/_memimporter.c
patching file source/python-dynload.c
patching file source/start.c

OK, so we can build now - there is still a deprecation warning, but the build will succeed:

$ python3 -m build --wheel --skip-dependency-check --no-isolation
* Building wheel...
D:/msys64/tmp/py2exe_git/py2exe_setuptools.py:10: SetuptoolsDeprecationWarning: dep_util is Deprecated. Use fu
nctions from setuptools.modified instead.
!!

        ********************************************************************************
        Please use `setuptools.modified` instead of `setuptools.dep_util`.

        By 2024-May-21, you need to update your project and remove deprecated calls
        or your builds will no longer be supported.

        See https://github.com/pypa/setuptools/pull/4069 for details.
        ********************************************************************************

!!
  from setuptools.dep_util import newer_group
running bdist_wheel
...
adding 'py2exe-0.13.0.1.dist-info/RECORD'
removing build/bdist.mingw_x86_64/wheel
Successfully built py2exe-0.13.0.1-cp311-cp311-mingw_x86_64.whl

Right - so the trick here, is that an _memimporter.pyd file did NOT build, only an .o file:

$ find . -name '_mem*'
./build/temp.mingw_x86_64-cpython-311/source/_memimporter.o
./source/_memimporter.c

If we inspect the build log, we'll see that this .o file gets built directly into run-py311-mingw_x86_64.exe:

gcc -s ... build/temp.mingw_x86_64-cpython-311/source/_memimporter.o ... -lpython3.11 -o build/lib.mingw_x86_64-cpython-311/py2exe/run-py311-mingw_x86_64.exe -m64 -mconsole -municode

... which is probably why it is impossible to extract a _memimporter.pyd from the .exe.

So, now we have .o file, but we need .pyd - I found a hint in How to create a .pyd file? (thankfully, also with a reference to MSYS2/MINGW64) - so eventually, that command is:

gcc -shared -L/mingw64/bin \
./build/temp.mingw_x86_64-cpython-311/source/_memimporter.o \
build/temp.mingw_x86_64-cpython-311/source/memorymodule.o \
build/temp.mingw_x86_64-cpython-311/source/myloadlibrary.o \
build/temp.mingw_x86_64-cpython-311/source/actctx.o \
build/temp.mingw_x86_64-cpython-311/source/python-dynload.o \
-lpython3.11 -o _memimporter.pyd

So, now we have a _memimporter.pyd; and to have it available so the test script will work, we copy it to /mingw64/lib/python3.11/site-packages/ (for some reason, copying it to /mingw64/lib/python3.11/site-packages/py2exe does not work):

$ cp -av /tmp/py2exe_git/_memimporter.pyd /mingw64/lib/python3.11/site-packages/
'/tmp/py2exe_git/_memimporter.pyd' -> '/mingw64/lib/python3.11/site-packages/_memimporter.pyd'

Then we can finally test the import and see no errors:

$ python3 -c 'import _memimporter'
$

And now we can run the test script from the OP - however, for some reason, I do not get printouts for _socket:

$ python3 test_zipextimporter.py
$

... but at least there are no errors related to _memimporter.