Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

How to install data #678

Answered by dnicolodi
peekxc asked this question in Q&A
Oct 6, 2024 · 1 comments · 13 replies
Discussion options

It is mentioned here that one limitation of meson-python is no data--via meson's install_data function---is not available, though it hints that you can perhaps still include data inside a Python module.

How? It is entirely opaque how to proceed.

For example, if I have the layout:

├── pyproject.toml
├── src
│   └── my_package
│       ├── __init__.py
│       ├── data
│       │   └── __init__.py
|       |   ├── meson.build
│       └── meson.build

What do I need to put in either of the meson.build files to use importlib.resources here?

You must be logged in to vote

What you need is:

py = import('python').find_installation(pure: false)
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir() / 'my_package/data',
)

or the equivalent

py = import('python').find_installation()
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir(pure: false) / 'my_package/data',
)

Replies: 1 comment · 13 replies

Comment options

There are just Python source files in the package layout in your example. Which data files do you want to install?

You must be logged in to vote
13 replies
@dnicolodi
Comment options

I'll be happy to do it, but I don't have access to the project on readthedocs... @rgommers can you please give me access?

@peekxc
Comment options

That's the documentation for a very old version of meson-python.

Ahh, thank you, I didn't realize I was reading from an older doc site.

The error message explains clearly what's wrong. Just follow the guidance in the error message on how to fix it.

Not quite. The error message recommends setting pure: false, but this makes no sense as this is not an option in install_data; I have already set this in the find_installation call, i.e.:

py = py_mod.find_installation(pure: true, required: true)

Semantically, the pure parameter is apparently there to allow for the placement of platform-dependent files (e.g. *.so) alongside source *.py files. But I need data files, not just python source files and platform binaries.

I was able to finally get the data included in the wheel by treating the data files as python source files and using py.install_sources, i.e.

py.install_sources(
	meson.project_source_root() / 'src' / 'my_package' / 'data' / 'aggregation.npz',
	subdir: pkg_name / 'data',
	pure: false       # Will be installed next to binaries
)

This keeps it in the wheel and lets me import with importlib, though I haven't tested it on all platforms. The docs should prob be changed to mention this, as install_data seems for me to be a red herring.

@dnicolodi
Comment options

The pure: false is an option for py.get_install_dir().

py = py_mod.find_installation(pure: true, required: true)

This is wrong if you are installing extension modules.

Your solution is semantically wrong.

@dnicolodi
Comment options

What you need is:

py = import('python').find_installation(pure: false)
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir() / 'my_package/data',
)

or the equivalent

py = import('python').find_installation()
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir(pure: false) / 'my_package/data',
)
Answer selected by peekxc
@eli-schwartz
Comment options

Semantically, the pure parameter is apparently there to allow for the placement of platform-dependent files (e.g. *.so) alongside source *.py files. But I need data files, not just python source files and platform binaries.

It is semantically there to semantically delineate the wheel (*.whl) file format's concept of purelib and platlib data, where some of that data may be files named *.py and possibly even files named *.so.

Python imports operate at the level of directory entries and you cannot have two directories named the same thing, one in the purelib part of sys.path and one in the platlib part of sys.path, without juggling the intricacies of namespace packages. This is a problem regardless of whether you are installing .py files or .npz files or .txt files.

The split between purelib and platlib is an artifact of a handful of distributors who want to put compiled C extensions in /usr/lib/python3.12/site-packages but put pure python modules in /use/share/python3.12/site-packages in order to conform to their own policies about "multilib" installations (32-bit software running on a 64-bit OS) where architecture-independent files are only installed once but compiled C code is installed once for 32-bit and once for 64-bit. It's kind of a weird appendage but everyone is forced to know about it -- in general you want your entire package to be either one or the other but if you have multiple top-level imports it is technically possible to split them across both installation schemes.

@peekxc
Comment options

This is wrong if you are installing extension modules.

Ahh, I guess I misunderstood the purpose of the pure flag, which is what caused this issue (I am installing extension modules). Changing all values to pure: false seems to have fixed the issue.

Sorry for wasting your time

@peekxc
Comment options

The split between purelib and platlib is an artifact of a handful of distributors who want to put compiled C extensions in /usr/lib/python3.12/site-packages but put pure python modules in /use/share/python3.12/site-packages in order to conform to their own policies about "multilib" installations (32-bit software running on a 64-bit OS) where architecture-independent files are only installed once but compiled C code is installed once for 32-bit and once for 64-bit. It's kind of a weird appendage but everyone is forced to know about it -- in general you want your entire package to be either one or the other but if you have multiple top-level imports it is technically possible to split them across both installation schemes.

Ahh I see. I assume the former usecase for placing something in /usr/lib/../site-packages is for larger often shared binaries (like blas?). In my case I have pythranized modules so keeping them alongside the original python source files is preferable.

@eli-schwartz
Comment options

No, it's literally because C extensions, even if they are only 15kb, semantically belong in the "libdir" for FHS reasons, and files such as ASCII / unicode interpreted script files, which do not care which bitness your processor is nor which operating system syscall ABI is used, are permitted in "datadir".

  • binaries are architecture-dependent, the FHS says: put it in libdir.
  • .py files, and .npz files, are architecture-agnostic, the FHS says: you may feel free to put them in datadir if you please, though if they have to maintain directory structure with binaries then you may prefer to put them in libdir.

Everyone must put python "platlib" (architecture-dependent) in /usr/lib/python3.12/site-packages, because that is the only valid location.

Most distributors also put "purelib" (architecture-agnostic) in /usr/lib/python3.12/site-packages, because it is allowed and has the advantage of being straightforward and simple.

Some put "purelib" (architecture-agnostic) in /usr/share/python3.12/site-packages, because... they like doing it? They have the right to do it according to the spec and I guess it allows them to save installation disk space when you have a 32-bit and a 64-bit python installed side by side, since you can install 3 packages:

  • 32-bit binaries
  • 64-bit binaries
  • architecture-agnostic .py files

and have the first and second each depend on the third.

If they put .py files alongside the binaries then they'd have to install the .py files twice, once for 32-bit and once for 64-bit.

They are not using pip as a package manager, that is for sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
3 participants
Converted from issue

This discussion was converted from issue #677 on October 06, 2024 19:27.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.