Failed to build Cython example with VS2017

[Zhou-Ao]

When I try to build the cython example in https://github.com/zeromq/pyzmq/tree/master/examples/cython, I find the following link error.

running build_ext
building 'cyzmq_example' extension
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\azhou-p\AppData\Local\Continuum\anaconda3\lib\site-packages -IC:\Users\azhou-p\AppData\Local\Continuum\anaconda3\lib\site-packages\zmq\utils -IC:\Users\azhou-p\AppData\Local\Continuum\anaconda3\lib\site-packages\zmq\include -IC:\Users\azhou-p\AppData\Roaming\Python\Python37\site-packages\numpy\core\include -IC:\Users\azhou-p\AppData\Local\Continuum\anaconda3\include -IC:\Users\azhou-p\AppData\Local\Continuum\anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /Tccyzmq.c /Fobuild\temp.win-amd64-3.7\Release\cyzmq.obj
cyzmq.c
c:\users\azhou-p\appdata\roaming\python\python37\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\azhou\AppData\Local\Continuum\anaconda3\Library\lib /LIBPATH:C:\Users\azhou-p\AppData\Local\Continuum\anaconda3\libs /LIBPATH:C:\Users\azhou-p\AppData\Local\Continuum\anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64" /EXPORT:PyInit_cyzmq_example build\temp.win-amd64-3.7\Release\cyzmq.obj /OUT:D:\QIO\librairies\azhou-p\rpc\pyzmq_cython\cyzmq_example.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\cyzmq_example.cp37-win_amd64.lib
   Creating library build\temp.win-amd64-3.7\Release\cyzmq_example.cp37-win_amd64.lib and object build\temp.win-amd64-3.7\Release\cyzmq_example.cp37-win_amd64.exp
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_ctx_destroy
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_socket
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_close
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_connect
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_ctx_new
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_msg_init_size
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_msg_data
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_msg_send
cyzmq.obj : error LNK2001: unresolved external symbol __imp_zmq_msg_close
D:\QIO\librairies\azhou-p\rpc\pyzmq_cython\cyzmq_example.cp37-win_amd64.pyd : fatal error LNK1120: 9 unresolved externals
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120

I am using MSVC build tools 2017 14.16.27023 with the libzmq.dll complied with "pip install pyzmq". The pyzmq version is 20.0.0.

Any thoughts will be much appreciated. Thanks in advance.

[minrk]

I've only tested this with linux/mac, and Windows linking is substantially different. I don't have access to a Windows environment to test with, but I suspect that it needs to link libzmq directly. Try something like:

extensions = [
    Extension(
        "cyzmq_example",
        ["cyzmq.pyx"],
        libraries=["libzmq"],
        library_dirs=zmq.get_library_dirs(),
        include_dirs=zmq.get_includes() + [numpy.get_include()],
    )
]

Alternative: linux/mac Python compilation uses a version of -undefined dynamic_lookup by default. Maybe there's an equivalent msvc linker flag we can be passing? importing zmq means that it will always be loaded, so telling the linker that it's okay might be enough.

If you can get that to work, it would be great to update the example with Windows instructions!

[Zhou-Ao]

Hi minrk,

Thanks very much for your prompt reply.

I have tried that the direct link to libzmq with libraries=["libzmq"] doesn't work. And I am still investigating how to set the equivalent linker flag of dynamic_lookup with msvc.

I just tested the compilation under linux ubuntu 20.04 with gcc. The exetension building went thought. However, when I tred to import functions from cyzmq_example. The symbols under libzmq were still showing undefined. I am not sure if the issues under window and linux are related.

running build_ext
building 'cyzmq_example' extension
gcc -pthread -B /home/azhou/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/azhou/anaconda3/lib/python3.7/site-packages -I/home/azhou/anaconda3/lib/python3.7/site-packages/zmq/utils -I/home/azhou/.local/lib/python3.7/site-packages/numpy/core/include -I/home/azhou/anaconda3/include/python3.7m -c cyzmq.c -o build/temp.linux-x86_64-3.7/cyzmq.o
In file included from /home/azhou/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832,
                 from /home/azhou/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/azhou/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from cyzmq.c:633:
/home/azhou/.local/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gcc -pthread -shared -B /home/azhou/anaconda3/compiler_compat -L/home/azhou/anaconda3/lib -Wl,-rpath=/home/azhou/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/cyzmq.o -o /home/azhou/qio/libraries/zmq_cython/cyzmq_example.cpython-37m-x86_64-linux-gnu.so

Traceback (most recent call last):
  File "example.py", line 8, in <module>
    from cyzmq_example import cython_sender, mixed_receiver
ImportError: /home/azhou/qio/libraries/zmq_cython/cyzmq_example.cpython-37m-x86_64-linux-gnu.so: undefined symbol: zmq_socket

[minrk]

I've looked into this one a bit, and discovered that mac/linux are indeed affected by the same issue if zmq is not imported first, i.e. they rely on the fact that pyzmq loads the symbols from the libzmq dylib before being imported. Linking libzmq directly would be a good idea, but is very hairy for bundled libzmq as discussed in #1412, especially on macOS with install_name shenanigans. So the experience is better to not link zmq directly if we can rely on lazy loading in pyzmq (for a 'real' project, you can safely import zmq first in a top-level Python package before your extension modules are loaded).

EDIT:
Can you add libraries=["libzmq.pyd"] or similar and see if that works on Windows? My guess is that libraries=["libzmq"] doesn't work due to the fact that the name isn't exactly libzmq.dll. Check the exact path of the dll and see if you can load it by abspath, either in libraries or perhaps extra_objects

[Zhou-Ao]

Hi minrk,

Thanks for the suggestions.

I've figured out how to build under windows with msvs. There are two things to add in the extension setup.

1. As you suggested, add libraries=["libzmq"] (without any file extension).
2. Add library_dirs=['%PYTHON_PATH%/Library/lib'] which is the directory contains libzmq.dll. And it seems cython doesn't find .dll under sys.path, as I initially failed to build with the directory in %PATH% only.