Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Ctypes and editable mode #653

Answered by keturn
peppedilillo asked this question in Q&A
Jul 31, 2024 · 2 comments · 2 replies
Discussion options

Dears. Using pip install -e . --no-build-isolation, as suggested by the docs, I can correctly install the package and compile my C code, which is called through ctypes. However once I try to use the package it fails to find the shared libraries:

  File "/home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/bft_cwrap.py", line 16, in <module>
    clib_bft = ctypes.CDLL(_LIBCFOCUS)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/.sharedlibs/lib-pfocus.so: cannot open shared object file: No such file or directory

The _LIBCFOCUS variable above is implemented here:

def _library_path(libname: str) -> str:
    """Gets the path to the C shared library, with appropriate extension.
    Assumes the library and this file to be installed in the same directory"""
    from pathlib import Path
    import sys

    if sys.platform.startswith("win32"):
        suffix = ".dll"
    elif sys.platform.startswith("linux"):
        suffix = ".so"
    elif sys.platform.startswith("darwin"):
        suffix = ".dylib"
    else:
        raise OSError("System not supported")

    return str(Path(__file__).parent / f"{libname}{suffix}")


_LIBCFOCUS = _library_path(".sharedlibs/lib-pfocus")

If you want to take a look, here is the software I'm working on.

The problem is that, once in editable mode, the code looks for the shared libraries in the wrong place. While not in editable mode, the libraries are installed in my local environment directory (site-packages/hbstools/triggers/.sharedlibs/lib-pfocus.so). However, once in editable-mode, I do not know where they are actually installed, but they are expected to be in my source directory.

I have asked this question under an issue (sorry @rgommers for reviving a dead issue) and got a kind reply from @eli-schwartz, which suggested to install the shared libraries in multiple places, so that I can try to import from these. However it is not fully clear to me how this should be carried out, and if there is any better solution not requiring multiple copies of the same file.

Is it possible to install my project in editable mode?

Thank you very much.

You must be logged in to vote

I think the trick here is to use importlib.resources instead of __file__:

importlib.resources.files(__package__).joinpath(".sharedlibs/libp-focus.so")

meson-python's editable-mode hooks will resolve that to the build directory.

Replies: 2 comments · 2 replies

Comment options

I think the trick here is to use importlib.resources instead of __file__:

importlib.resources.files(__package__).joinpath(".sharedlibs/libp-focus.so")

meson-python's editable-mode hooks will resolve that to the build directory.

You must be logged in to vote
2 replies
@eli-schwartz
Comment options

importlib-resources is totally unrelated to the challenges of finding files in an editable install, using a different source/build layout than a "real" install would have.

@rgommers
Comment options

I don't think it is, this answer looks correct to me. The editable install works through importlib hooks, and importlib.resources is the best way to load files. This is what I needed to do in SciPy as well to load generated files: scipy/scipy@2ddb28b.

Answer selected by rgommers
Comment options

I have asked this question under an issue (sorry @rgommers for reviving a dead issue) and got a kind reply from @eli-schwartz, which suggested to install the shared libraries in multiple places, so that I can try to import from these. However it is not fully clear to me how this should be carried out, and if there is any better solution not requiring multiple copies of the same file.

I think I had suggested at the time, not that you install the libraries in multiple places, but rather that your _LIBCFOCUS variable should look in multiple places.

The library is created in hbstools/triggers/meson.build

You look for the library in hbstools/triggers/__init__.py

That means that the library itself may be in one of two places:

  • installed, non-editable, next to that __init__.py, in ./.sharedlibs/{libname}{suffix}
  • inside the meson build directory, at {build_directory}/hsbtools/triggers/{libname}{suffix}

The trick is going to be locating {build_directory} so you can look at that instead. The "easiest" way is if __file__ for your editable install is pointing to {build_directory}/hsbtools/triggers/__init__.py.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
4 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.