Simple copying of windows DLLs in a wheel #629

May 15, 2024

kpenev
May 15, 2024

I am trying to create windows wheels for a python project that includes a shared library which in turn depends on a compiled boost library. The library is only used through cdll.LoadLibrary so there is no need to fix rpaths like for an extension module for example. I simply need the internal and boost shared libraries to both be copied into the wheel. However, I do not see an option to accomplish this. Instead meson-python crashes at the end with a message:

Bundling libraries in wheel is not supported on win32

I am guessing there is a way to do this since it is mentioned in the documentation for example that users can manually invoke delvewheel and in a few discussion threads I found, but I could not figure out how to do it. Any help will be much appreciated.

I should mention that I tried adding to pyproject.toml:

[tool.cibuildwheel.windows]
repair-wheel-command = ""

But that did not seem to disable the above error.

Answered by dnicolodi

May 17, 2024

meson-python assumes that anything installed in {libdir} is a shared library. If you are using Eigen as a dependency, you need to make sure that it is not installed as part of the Python wheel: it does not make sense to distribute a Python wheel containing an headers only library. I think that most likely the meson.build for Eigen in wrapdb should not install the headers in the first place, but that is another issue. You can pass the --skip-subprojects=eigen option to meson install adding

[tool.meson-python.args]
install = ['--skip-subprojects=eigen']

to pyproject.toml.

View full answer

rgommers · May 16, 2024

rgommers
May 16, 2024
Maintainer

This should be possible indeed. If all you're doing is building a shared library and want to install it inside your package, you can use the install: true and install_dir: keywords to shared_library like so: https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/scipy/special/meson.build#L28-L34

The Boost library is different, since that's external. delvewheel should take care of that vendoring. Again an example from SciPy, which uses delvewheel: https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/pyproject.toml#L150

But that did not seem to disable the above error.

The error is about the internal library, so unrelated to delvewheel.

If you run into more trouble, please feel free to point us at your source code.

3 replies

rgommers May 16, 2024
Maintainer

Also see gh-551 for an in-progress PR to improve the handling of internal shared libraries on Windows.

kpenev May 16, 2024
Author

Thank you for the prompt response.

I already have my shared library defined as:

shared_library(
    'superphot', 
    cpp_src, 
    dependencies: dependencies,
    install: true,
    install_dir: py.get_install_dir() / 'superphot'
)

And that indeed does what it is supposed to do (i.e. copy the library next to the python code that uses it. The problem arises when buld_wheel is invoked (in my case by cibuildwheel) which then promptly crashes. I am assuming there is a way to disable the call to mesonpy._rpath.fix_rpath on the internally built library. Or is the method perhaps to build the library as a separate pre-build step and then just list it as an input? The full traceback I get (if it helps) is:

    Traceback (most recent call last):
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
        main()
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 152, in prepare_metadata_for_build_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 1020, in wrapper
        return func(*args, **kwargs)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 1074, in build_wheel
        return project.wheel(out).name
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 925, in wheel
        return builder.build(directory)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 481, in build
        self._install_path(whl, src, dst)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 440, in _install_path
        mesonpy._rpath.fix_rpath(origin, libspath)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\_rpath.py", line 64, in fix_rpath
        raise NotImplementedError(f'Bundling libraries in wheel is not supported on {sys.platform}')
    NotImplementedError: Bundling libraries in wheel is not supported on win32

If it helps, my full meson.build file is:

#Control file for meson/meson-python (compiling and PIP packaging)

project(
    'superphot', 
    'cpp',
    version: '0.0.0',
    license: '<++>',
    meson_version: '>= 1.4.0',
    default_options: [
        'optimization=3', 
        'cpp_std=c++14', 
        'prefer_static=true'
    ],
)

py = import('python').find_installation(pure: false)

compiler_id = meson.get_compiler('cpp').get_id()
toolchain = {
    'gcc': 'TOOLCHAIN_GCC', 
    'clang': 'TOOLCHAIN_CLANG',
    'msvc': 'TOOLCHAIN_MSVC'
}[compiler_id]

add_project_arguments('-D' + toolchain,
                      '-DTRACK_PROGRESS', 
                      language : 'cpp')
if get_option('buildtype') == 'debug'
    add_project_arguments('-DVERBOSE_DEBUG', '-DDEBUG', language: 'cpp')
elif get_option('buildtype') == 'release'
    add_project_arguments('-DNDEBUG', language: 'cpp')
endif

dependencies = [
    dependency('boost', modules: ['program_options']),
    dependency('eigen3')
]


if host_machine.system() == 'windows'
    add_project_arguments(
      '-DBOOST_RESULT_OF_USE_TR1', 
      '-D_USE_MATH_DEFINES',
      '-DNO_VISIBILITY',
      language: 'cpp'
    )
else
    add_project_arguments(
      '-fvisibility=hidden', 
      '-DNIX_VISIBILITY',
      language : 'cpp'
    )
endif

cpp_src = [
   ... long list of cpp files ...
]

shared_library(
    'superphot', 
    cpp_src, 
    dependencies: dependencies,
    install: true,
    install_dir: py.get_install_dir() / 'superphot'
)

subdir('PythonPackage/superphot')

rgommers May 16, 2024
Maintainer

I am assuming there is a way to disable the call to mesonpy._rpath.fix_rpath on the internally built library.

Looking at the code, fix_rpath is only called if _WheelBuilder._has_internal_libs is True, which in turn is the case if {libdir} or libdir_shared} are encountered in build/meson-info/intro-install_plan.json. Could you check that file to see what target has one of those locations? Fixing that should resolve the problem.

For the SciPy example I linked, the metadata in intro-install_plan.json is:

"/path/to/build/scipy/special/libsf_error_state.so": {"destination": "{py_platlib}/scipy/special/libsf_error_state.so", "tag": "runtime", "subproject": null},

dnicolodi · May 17, 2024

kpenev
May 17, 2024
Author

That clarifies things a lot, though still not sure how to get it to work.

The intro-install_plan.json file is too big to usefully quote here directly (121K), so I am attaching it (intro-install_plan.json) if the excerpt below is not enough. But it seems the issue is due to the fact that building our library requires the header only project Eigen, which we pull during the build from wrapdb. The relevant entry in intro-install_plan.json is:

"data": {"D:\\a\\SuperPhot\\SuperPhot\\build\\meson-private\\eigen3.pc": {"destination": "{libdir}\\pkgconfig\\eigen3.pc", "tag": "devel", "subproject": "eigen"}}

Since Eigen is header only, the resulting library does not need any @rpath to be fixed, but I guess mesonpy doesn't know that.

1 reply

dnicolodi May 17, 2024
Maintainer

meson-python assumes that anything installed in {libdir} is a shared library. If you are using Eigen as a dependency, you need to make sure that it is not installed as part of the Python wheel: it does not make sense to distribute a Python wheel containing an headers only library. I think that most likely the meson.build for Eigen in wrapdb should not install the headers in the first place, but that is another issue. You can pass the --skip-subprojects=eigen option to meson install adding

[tool.meson-python.args]
install = ['--skip-subprojects=eigen']

to pyproject.toml.

Answer selected by kpenev

rgommers · May 17, 2024

kpenev
May 17, 2024
Author

Thank you very much. That worked beautifully. It would make sense by default to not install libraries which have nothing that needs to be included in the package, but I suspect removing install by default would run afoul of some of the intended uses of wrapdb. In any case the solution is rather clean in the end.

3 replies

rgommers May 17, 2024
Maintainer

but I suspect removing install by default would run afoul of some of the intended uses of wrapdb

I think so too, so I don't expect anything to change here.

That worked beautifully.

Awesome, glad that it's resolved for you. Would you mind hitting Mark as Answer on whatever you think is the most appropriate comment?

kpenev May 17, 2024
Author

Sorry. I did that now. I saw the button to close the discussion, but did not see the mark as answer button.

rgommers May 17, 2024
Maintainer

No worried at all, thanks @kpenev

WillAyd · Aug 5, 2024

WillAyd
Aug 5, 2024

I'm running into this same issue, but actually need to install the subproject library into my Python package. The subproject I am using is nanoarrow from the wrapdb, whose configuration has an install: true default.

I read through the discussion but I'm not sure what to do since I do need to bundle nanoarrow into my library (a static lib would also be an option):

The default meson configuration for a windows run on cibuildwheel yielded:

NotImplementedError: Bundling libraries in wheel is not supported on win32

Though macOS and ubuntu were fine.

I tried using static linkage in my meson configuration via:

nanoarrow_dep = dependency('nanoarrow', default_options: ['default_library=static'])

which in turn yielded this error from meson-python (all platforms failed):

meson-python: error: Could not map installation path to an equivalent wheel directory: '{libdir_static}\\libnanoarrow.a'

After seeing issue #525 I tried reverting back to shared linkage while pinning meson-python 0.13.2 but ran into the same issue as the OP:

NotImplementedError: Bundling libraries in wheel is not supported on win32

I tried adding a delvewheel call to the shared linkage + version pin:

[tool.cibuildwheel.windows]
before-build = "pip install delvewheel"
repair-wheel-command = "delvewheel repair --add-path {libdir} -w {dest_dir} {wheel}"

But during the repair phase I get:

FileNotFoundError: Unable to find library: libnanoarrow.dll

Is there something simple I am overlooking here? Unfortunately I don't have access to a PC so have been trying this all from GHA; I'm assuming the delvewheel approach is the closest?

If it helps this is a PR showing a lot of the things I've walked through

WillAyd/pandas-bitmask#1

2 replies

WillAyd Aug 5, 2024

FWIW I would probably prefer statically linking the libraries (if that can solve the windows issue), and I see that meson-python should forward skip-subprojects, but doing that from the CLI via pip install . --config-settings=install-args="--skip-subprojects" or setting this in pyproject.toml:

[tool.meson-python.args]
install = ['--skip-subprojects']

seems to have no effect

WillAyd Aug 5, 2024

Ah ignore the above comment - I had to remove the meson-python 0.13.2 pin in pyproject.toml to get the skipping subprojects to work during install.

So I can get the static libs working on non-Windows builds, but get a runtime error with Windows with that same configuration:

  E   ImportError: DLL load failed while importing <the_library>: The specified module could not be found.

rgommers · Dec 6, 2024

tobiasdiez
Dec 6, 2024

I'm experiencing this issue as well. But instead of worrying about how to distribute the wheels correctly, I actually "only" want to install the package including the subproject in the current venv.

So just have pip install . from the source directory of the project work. This (sometimes?) builds a wheel first and then installs it. But building the wheel fails if a subproject is necessary to build and run the project. Any advice on how to make it properly work on Windows?

Workaround: use editable install, which doesn't build a wheel first.

1 reply

rgommers Dec 6, 2024
Maintainer

pip install . will always build a wheel first, then install it.

The docs being added in gh-700 may help address some of the questions in this discussion, including this one. It also has a new test package with a library in a subproject.

Search code, repositories, users, issues, pull requests...

Simple copying of windows DLLs in a wheel #629

Uh oh!

Uh oh!

kpenev May 15, 2024

Replies: 5 comments · 10 replies

Uh oh!

Uh oh!

rgommers May 16, 2024 Maintainer

Uh oh!

rgommers May 16, 2024 Maintainer

Uh oh!

kpenev May 16, 2024 Author

Uh oh!

rgommers May 16, 2024 Maintainer

Uh oh!

kpenev May 17, 2024 Author

Uh oh!

dnicolodi May 17, 2024 Maintainer

Uh oh!

kpenev May 17, 2024 Author

Uh oh!

rgommers May 17, 2024 Maintainer

Uh oh!

kpenev May 17, 2024 Author

Uh oh!

rgommers May 17, 2024 Maintainer

Uh oh!

WillAyd Aug 5, 2024

Uh oh!

WillAyd Aug 5, 2024

Uh oh!

WillAyd Aug 5, 2024

Uh oh!

tobiasdiez Dec 6, 2024

Uh oh!

rgommers Dec 6, 2024 Maintainer

kpenev
May 15, 2024

rgommers
May 16, 2024
Maintainer

rgommers May 16, 2024
Maintainer

kpenev May 16, 2024
Author

rgommers May 16, 2024
Maintainer

kpenev
May 17, 2024
Author

dnicolodi May 17, 2024
Maintainer

kpenev
May 17, 2024
Author

rgommers May 17, 2024
Maintainer

kpenev May 17, 2024
Author

rgommers May 17, 2024
Maintainer

WillAyd
Aug 5, 2024

tobiasdiez
Dec 6, 2024

rgommers Dec 6, 2024
Maintainer