• What is then supposed to happen is that building a wheel completes. However, vendoring external dependencies into a wheel is not the job of meson-python but rather of auditwheel/delocate/delvewheel.

Okay, yes I see that. There are other cases than cibuildwheel though. It would be really nice to use the meson subprojects feature or similar to be able to manage and build the dependencies when building locally from sdist or VCS. For cibuildwheel I would want to build GMP in the CIBW_BEFORE_ALL stage though before mesonpy gets called. Ideally the build configuration could somehow be shared for all of these things.

I don't think this is the case, unless @oscarbenjamin passes a buildidr to meson-python: the build directory where the subproject is gone once the wheel is build, thus the extension has an RPATH pointing to the location of the subproject build directory, but the library there is long gone when the extension is loaded. Indeed, for including the library in the wheel we would need to treat it as a local library. For doing that, it seems that what is missing is being able to map the installation path of the parts of the subproject into the wheel locations.

It is linking dynamically but I have GMP installed on the system:

$ ldd meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffce7b6f000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007facf2ee2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007facf2c00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007facf2f7f000)

$ ls -l /lib/x86_64-linux-gnu/libgmp.so.10
lrwxrwxrwx 1 root root 16 Apr  5 21:00 /lib/x86_64-linux-gnu/libgmp.so.10 -> libgmp.so.10.4.1

I was getting confused by the fact that libgmpy3-dev is needed in order to build the wheel but at runtime there is already a libgmp.so even if I uninstall the -dev files.

The libgmp3-dev package (there is a typo in the package name in your comment) is required for the C header to be available for compilation. Most distributions split libraries in runtime and development library packages, where the -dev package essentially contains the static library version of the library and the header files.

So how does the linker end up picking the system gmp? It is a bit worrying that we've built a local GMP and then ended up linking to the system one.

During configuration meson can tell that the GMP headers are not installed although it reports this as a "run-time dependency":

Run-time dependency gmp found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency gmp

Executing subproject gmp 

It therefore decides to download and build GMP:

$ meson setup build --wipe
The Meson build system
Version: 1.1.0
Source dir: /home/oscar/current/active/tmp/mesontest
Build dir: /home/oscar/current/active/tmp/mesontest/build
Build type: native build
Project name: meson-test
Project version: undefined
C compiler for the host machine: cc (gcc 11.3.0 "cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0")
C linker for the host machine: cc ld.bfd 2.38
Cython compiler for the host machine: cython (cython 0.29.34)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/home/oscar/.pyenv/versions/3.11.3/envs/mesontest-3.11/bin/python3.11)
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Found CMake: /usr/bin/cmake (3.22.1)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency gmp found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency gmp

Executing subproject gmp 

gmp| Project name: gmp
gmp| Project version: 6.2.1
gmp| C compiler for the host machine: cc (gcc 11.3.0 "cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0")
gmp| C linker for the host machine: cc ld.bfd 2.38
gmp| subprojects/gmp-6.2.1/meson.build:7: WARNING: Module External build system has no backwards or forwards compatibility and might not exist in future releases.
gmp| Program /home/oscar/current/active/tmp/mesontest/subprojects/gmp-6.2.1/configure found: YES (/home/oscar/current/active/tmp/mesontest/subprojects/gmp-6.2.1/configure)
gmp| Program make found: YES (/usr/bin/make)
gmp| External project gmp-6.2.1: configure
gmp| Build targets in project: 1
gmp| Subproject gmp finished.

Dependency gmp found: YES 6.2.1 (overridden)
Build targets in project: 2

meson-test undefined

  Subprojects
    gmp: YES 1 warnings

Found ninja-1.10.1 at /usr/bin/ninja

Then when I actually build it links against the system installed GMP even though we just built a local GMP (and they are not guaranteed to be compatible):

$ ninja -v -C build
ninja: Entering directory `build'
[0/4] /home/oscar/.pyenv/versions/mesontest-3.11/bin/meson --internal externalproject --name gmp-6.2.1 --srcdir /home/oscar/current/active/tmp/mesontest/subprojects/gmp-6.2.1 --builddir /home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/build --installdir /home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/dist --logdir /home/oscar/current/active/tmp/mesontest/build/meson-logs --make /usr/bin/make subprojects/gmp-6.2.1/gmp-6.2.1.stamp subprojects/gmp-6.2.1/gmp-6.2.1.d
[2/4] cython -M --fast-fail -3 /home/oscar/current/active/tmp/mesontest/meson_test/_meson_test.pyx -o meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c
[3/4] cc -Imeson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p -Imeson_test -I../meson_test -Isubprojects/gmp-6.2.1 -I/home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/dist/usr/local/include -I/home/oscar/.pyenv/versions/3.11.3/include/python3.11 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -w -std=c99 -O2 -g -fPIC -MD -MQ meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -MF meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o.d -o meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -c meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c
[4/4] cc  -o meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -L/home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/dist/usr/local/lib/x86_64-linux-gnu -Wl,--start-group -lgmp -Wl,--end-group
$ ldd build/meson_test/_meson_test.cpython-311-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffe3b567000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f8161c1f000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8161800000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8161cbc000)

It is using -L/home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/dist/usr/local/lib/x86_64-linux-gnu and that directory contains the files:

$ ls /home/oscar/current/active/tmp/mesontest/build/subprojects/gmp-6.2.1/dist/usr/local/lib/x86_64-linux-gnu
libgmp.la  libgmp.so  libgmp.so.10  libgmp.so.10.4.1  pkgconfig

However the .so ends up linking against the absolute path /lib/x86_64-linux-gnu/libgmp.so.10 (not /usr/local/lib/... or some local path).

So how does the linker end up picking the system gmp? It is a bit worrying that we've built a local GMP and then ended up linking to the system one.

This is how dynamic linking works. The Python extension module knows that it needs symbols contained in a shared library named libgmp.so.10 and at most it has a list of locations where to look for it in addition to the standard locations configured for the shared library loader. All locations are probed in order and the first libgmp.so.10 found is loaded and dynamically linked to the extension module.

Simplifying a bit, for compiling you need the header files. In this case, Meson tries to find them using pkg-config and cmake. It does not find them and thus falls back to compiling the gmp subproject which will provide them. However, the libgmp.so.10 build as part of the subproject is not installed anywhere, thus when you load the Python extension module, the dynamic loader resolves the symbols via the system installed libgmp.so.10.

So far everything is working as expected. What does not work is that the subproject adds some files to the Meson installation manifest, and meson-python does not handle these correctly. It looks like something specific to external_subproject.

You need to decide is whether you want to include libgmp.so.10 in the Python wheel or require the users of your package to provide a system installed libgmp.so.10. Both are valid solution. Which one to prefer depends on the nature of your project and which kind of users you expect for the package.

If you expect the users of provide the library via a system dependency, I would drop the subproject fallback and require the development package to be installed to compile the Python wheel. Alternatively, you can still use the subproject fallback for getting what is required for the compilation, but exclude the content of the subproject from the installation. I don't know what is the easiest way to achieve that in the case of an external_subproject subproject. What is almost certain to work is to use instllation tags to filter the installed components via the tool.meson-python.args.intall setting in pyproject.toml.

Adding libgmp.so.10 into the wheel and adjusting the RPATH for the extension module to look for it in the right place is supported transparently by meson-python only on Linux ans macOS. If you need this to work on Windows, there is some manual setup required.

A third option is to link to libgmp.so.10 statically. In this case you just need to exclude the subproject from the installed components. For example as explained above.

From my perspective it would be fine if I needed to add some configuration somewhere that tells mesonpy which of these files I want to keep and where they should go.

I'm exploring updating pyzmq's build system to either meson or scikit-build-core, and I think this describes my use case exactly - find libzmq if it's around, download and build it if it's not. This is complicated by the fact that the latest libzmq release itself has added a dependency on libsodium, so I need to build two libraries. The hard part has been making sure it's available at runtime, especially for users who pip install from source, such as a platform that doesn't have a wheel (e.g. Python prerelease testers, some folks cross-compiling for phones I think).

Our solution thus far has been to build libzmq as a distutils Extension to inherit all the compiler configuration, w3hich has worked surprisingly well for many years, but this has all kinds of subtle, hairy consequences that I'd like to lose.

Hi @minrk, thanks for sharing your use case. We've made some progress here I think - or at least gained more experience on how to best do this - since the last post on this thread.

The main question I have for pyzmq now is whether, if libzmq is not already installed and you must download and build it, you have to use it as a shared library, or you can build it as a static library (libsodium too) and fold it into a Python extension module? The latter is simpler, because it avoids issues with things like Windows not having RPATH support.

static library might work for bundled libzmq/libsodium. An issue has been that it's used in quite a few extensions, which would make static linking complicated. But that's no longer the case, there is now only one Extension to build, so static linking may well be an option.

I'll need to learn quite a bit about how to make this work for Windows. libzmq/libsodium both use autotools on non-Windows, but for Windows libsodium has visual studio solutions while ZeroMQ uses CMake.

pyzmq is also a little complicated because the backend is compiled with Cython for CPython and CFFI for PyPy.

So I need something like:

if libzmq_found:
    locate libs, includes with pkg-config/etc. (standard, simple)
else:
    fetch libsodium, libzmq
    if Windows:
        build static libsodium with msbuild
        build static libzmq against static libsodium with cmake
    else:
        build static libsodium with autotools
        build static libzmq against static libsodium with autotools

if PyPy:
    _zmq = cffi extension
else:
    _zmq = cython extension

if libzmq_found:
    dynamically_link(_zmq, libzmq)
else:
    statically_link(_zmq, local libzmq)

Great, static libraries should make this easier. I think I can answer most of how to do the above, but probably this is the point to transfer this to a new discussion to keep this one for the shared library case. Would you mind copying this comment over?

pyzmq conversation continues in gh-556

I think though that what we need for that is a separate meson build configuration like having a subdirectory dependencies with a separate dependencies/meson.build file.

That'd be https://mesonbuild.com/Subprojects.html; subprojects can be built automatically before building the main project by meson-python.

The one thing that is missing I think is the auditwheel-like step, if you're getting a shared library dependency that itself depends on other shared libraries. I don't have time for a more well thought out response right now, but I think if this works:

python -m build --wheel  # including subprojects built automatically
auditwheel dist/*whl

and this does too:

pip install -e . --no-build-isolation  # works because rpath's not yet stripped

then you're not in such a bad place right?

the frontend creates an isolated environment and installs the Python build dependencies before invoking the build backend which is in mesonpy. The expectation in this setup is that any non-Python dependencies are installed "system-wide" before even the PEP 517 frontend is invoked.

I'll note that:

1. non-isolated builds are certainly a first class citizen, as they're used heavily for all distro packaging and for (e.g.) local development in a conda env
2. non system-wide native dependency installs work too on Linux and macOS, with the limitation on transitive rpaths you ran into. And on Windows it always requires delvewheel, because no RPATH support at all there.

I think if this works:

python -m build --wheel  # including subprojects built automatically
auditwheel dist/*whl

If we have that setup then what happens if a user installs from source from PyPI?

I am thinking of:

pip install --no-binary python-flint python-flint

The backend would build the libraries as subprojects but would not install them and would not bundle them in the wheel. Then the user is left with a broken install. I think that if the backend can't bundle the libraries then it is important to fail the build unless those libraries are already available externally.

From a development perspective we are in a good place but it is the user-install-from-source case that motivates the design I attempted in the OP. If we can't make that part work transparently then I think it is better if the mesonpy backend does not attempt to build the libs.

Users building from source when running pip install can be fraught with risk. The most blatant risk is on Windows where it's relatively unlikely that the user has any compiler installed at all, whether that's mingw or MSVC. Projects with Fortran code have it even worse...

I think that's part of why @rgommers and others are actually sort of hoping to make pip default to erroring out if there are wheels available for a package but no wheel available for your environment.

Depending on your use-case, a workaround could be linking the dependency statically by setting default_library=static.
But this breaks the use-cases where the dependency symbols should be found outside of the extension module itself.
EDIT: just found a way to bundle the dependency in the .whl, using custom_target(). See gh-556 for the details.