Converting Python Code to Cython: 'assert len(cubemap.shape) == 3' Compilation Error

Question

I took a Python code and needed to convert it to Cython to improve performance, but when I try to compile it, it gives me an error in this part of the code:

assert len(cubemap.shape) == 3

Here is my code:

import numpy
from . import utils

cimport numpy
cimport cython

cdef bytes mode_bilinear = b'bilinear'
cdef bytes mode_nearest = b'nearest'
cdef bytes cube_format_horizon = b'horizon'
cdef bytes cube_format_list = b'list'
cdef bytes cube_format_dict = b'dict'
cdef bytes cube_format_dice = b'dice'

@cython.boundscheck(False)
@cython.wraparound(False)
def c2e(numpy.ndarray[numpy.float64_t, ndim=3] cubemap, int h, int w, bytes mode=mode_bilinear, bytes cube_format=cube_format_dice):
    cdef int order

    if mode == mode_bilinear:
        order = 1
    elif mode == mode_nearest:
        order = 0
    else:
        raise NotImplementedError('unknown mode')

    if cube_format == cube_format_horizon:
        pass
    elif cube_format == cube_format_list:
        cubemap = utils.cube_list2h(cubemap)
    elif cube_format == cube_format_dict:
        cubemap = utils.cube_dict2h(cubemap)
    elif cube_format == cube_format_dice:
        cubemap = utils.cube_dice2h(cubemap)
    else:
        raise NotImplementedError('unknown cube_format')

    assert len(cubemap.shape) == 3
    assert cubemap.shape[0] * 6 == cubemap.shape[1]
    assert w % 8 == 0
    cdef int face_w = cubemap.shape[0]

    cdef numpy.ndarray[numpy.float64_t, ndim=2] uv = utils.equirect_uvgrid(h, w)
    cdef numpy.ndarray[numpy.float64_t, ndim=2] u = uv[:, 0]
    cdef numpy.ndarray[numpy.float64_t, ndim=2] v = uv[:, 1]
    cdef numpy.ndarray[numpy.float64_t, ndim=3] cube_faces = numpy.stack(numpy.split(cubemap, 6, 1), 0)

    # Get face id to each pixel: 0F 1R 2B 3L 4U 5D
    cdef numpy.ndarray[numpy.int32_t, ndim=2] tp = utils.equirect_facetype(h, w)
    cdef numpy.ndarray[numpy.float64_t, ndim=2] coor_x = numpy.zeros((h, w))
    cdef numpy.ndarray[numpy.float64_t, ndim=2] coor_y = numpy.zeros((h, w))

    for i in range(4):
        mask = (tp == i)
        coor_x[mask] = 0.5 * numpy.tan(u[mask] - numpy.pi * i / 2)
        coor_y[mask] = -0.5 * numpy.tan(v[mask]) / numpy.cos(u[mask] - numpy.pi * i / 2)

    mask = (tp == 4)
    cdef numpy.ndarray[numpy.float64_t, ndim=2] c = 0.5 * numpy.tan(numpy.pi / 2 - v[mask])
    coor_x[mask] = c * numpy.sin(u[mask])
    coor_y[mask] = c * numpy.cos(u[mask])

    mask = (tp == 5)
    c = 0.5 * numpy.tan(numpy.pi / 2 - numpy.abs(v[mask]))
    coor_x[mask] = c * numpy.sin(u[mask])
    coor_y[mask] = -c * numpy.cos(u[mask])

    # Final renormalize
    coor_x = (numpy.clip(coor_x, -0.5, 0.5) + 0.5) * face_w
    coor_y = (numpy.clip(coor_y, -0.5, 0.5) + 0.5) * face_w

    cdef numpy.ndarray[numpy.float64_t, ndim=3] equirec = numpy.stack([
        utils.sample_cubefaces(cube_faces[..., i], tp, coor_y, coor_x, order=order)
        for i in range(cube_faces.shape[2])
    ], axis=-1)

    return equirec

The error:

Compiling utils.pyx because it changed.
Compiling c2e.pyx because it changed.
[1/2] Cythonizing c2e.pyx
C:\Users\Ju-Bei\AppData\Local\Programs\Python\Python311\Lib\site-packages\Cython\Compiler\Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: C:\Users\Ju-Bei\Documents\Hyper Render 360\Hyper 360 Convert\fullcython\c2e.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...
    elif cube_format == cube_format_dice:
        cubemap = utils.cube_dice2h(cubemap)
    else:
        raise NotImplementedError('unknown cube_format')    

    assert len(cubemap.shape) == 3
                      ^
------------------------------------------------------------

c2e.pyx:37:22: Cannot convert 'npy_intp *' to Python object 
Traceback (most recent call last):
  File "C:\Users\Ju-Bei\Documents\Hyper Render 360\Hyper 360 Convert\fullcython\setup.py", line 5, in <module>
    ext_modules = cythonize([
                  ^^^^^^^^^^^
  File "C:\Users\Ju-Bei\AppData\Local\Programs\Python\Python311\Lib\site-packages\Cython\Build\Dependencies.py", line 1134, in cythonize    
    cythonize_one(*args)
  File "C:\Users\Ju-Bei\AppData\Local\Programs\Python\Python311\Lib\site-packages\Cython\Build\Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: c2e.pyx

Answer 1

It looks like there has been a recent change to the Cython API which must have broken h5py as fetched by pip. Cloning the master branch on my machine and compiling from it worked all right. Hopefully, the fixed code will make its way to the tarball that pip fetches from.

Answer 2

One way that Cython optimizes access to variables defined with a type numpy.ndarray[...] is that it replaces the .shape attribute with a pointer to the dimensions array:

See the code.

This makes looking up the shape a lot quicker, but it also means it isn't entirely Python compatible. A pointer doesn't have a "length" for example.

You probably just want to check the .ndim attribute anyway, and take it on trust that Numpy gets its arrays the right size.