Cross compiling pyzmq fails because the setup picks up the x86 compiler automatically

Question

I'm trying to cross compile the pyzmq library for my stm32mpu board and I'm facing the issue below.

/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'cffi_modules'
  warnings.warn(msg)
running bdist_wheel
running build
running build_py
running build_ext
running configure
{'libraries': ['zmq'], 'include_dirs': ['/home/Documents/arch_libzmq/include'], 'library_dirs': ['/home/Documents/arch_libzmq/lib'], 'runtime_library_dirs': ['/home/Documents/arch_libzmq/lib'], 'extra_link_args': []}
Warning: Skipping zmq version check
************************************************
building 'zmq.backend.cython._zmq' extension
arm-ostl-linux-gnueabi-gcc -mthumb -mfpu=neon-vfpv4 -mfloat-abi=hard -mcpu=cortex-a7 --sysroot=/home/Documents/STM/sdk/sysroots/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -O2 -pipe -g -feliminate-unused-debug-types -fPIC -DHAVE_SYS_UN_H=1 -I/home/Documents/arch_libzmq/include -Izmq/utils -I/usr/include/python3.10 -c zmq/backend/cython/_zmq.c -o build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o
cc1: warning: include location "/usr/include/python3.10" is unsafe for cross-compilation [-Wpoison-system-directories]
zmq/backend/cython/_zmq.c: In function ‘__pyx_pf_3zmq_7backend_6cython_4_zmq_5Frame_16set’:
zmq/backend/cython/_zmq.c:9675:12: warning: variable ‘__pyx_v_routing_id’ set but not used [-Wunused-but-set-variable]
 9675 |   uint32_t __pyx_v_routing_id;
      |            ^~~~~~~~~~~~~~~~~~
In function ‘__Pyx_ssize_strlen’,
    inlined from ‘__pyx_pf_3zmq_7backend_6cython_4_zmq_5Frame_14get’ at zmq/backend/cython/_zmq.c:9347:17,
    inlined from ‘__pyx_pw_3zmq_7backend_6cython_4_zmq_5Frame_15get’ at zmq/backend/cython/_zmq.c:9117:13:
zmq/backend/cython/_zmq.c:36018:18: warning: argument 1 null where non-null expected [-Wnonnull]
36018 |     size_t len = strlen(s);
      |                  ^~~~~~~~~
In file included from /usr/include/python3.10/Python.h:30,
                 from zmq/backend/cython/_zmq.c:35:
/home/Documents/STM/sdk/sysroots/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/usr/include/string.h: In function ‘__pyx_pw_3zmq_7backend_6cython_4_zmq_5Frame_15get’:
/home/Documents/STM/sdk/sysroots/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/usr/include/string.h:407:15: note: in a call to function ‘strlen’ declared ‘nonnull’
  407 | extern size_t strlen (const char *__s)
      |               ^~~~~~
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -O2 -pipe -g -feliminate-unused-debug-types build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o -L/home/Documents/arch_libzmq/lib -Wl,--enable-new-dtags,-R/home/Documents/arch_libzmq/lib -lzmq -o build/lib.linux-x86_64-3.10/zmq/backend/cython/_zmq.cpython-310-x86_64-linux-gnu.so
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: relocations in generic ELF (EM: 40)
/home/Documents/STM/sdk/sysroots/x86_64-ostl_sdk-linux/usr/bin/ld: build/temp.linux-x86_64-3.10/zmq/backend/cython/_zmq.o: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

If you see the logs, it first picks up the correct arm-ostl compiler and then again it picks up the x86 compiler below. I've compiled the libzmq and added the path in ZMQ_PREFIX. Also, here's my setup.cfg

[global]
skip_check_zmq = True

[build_ext]
; library_dirs =
; include_dirs =

I don't understand what am I doing wrong or missing here, any help is appreciated!

Edit 1 : I'm running this on x86_64 with Ubuntu 22.04