Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Testing application built with meson-python #659

Answered by eli-schwartz
voidtrance asked this question in Q&A
Discussion options

I am working on an application that is Python-based (so it has a bunch of Python files). However, the application also implements a Python extension and some shared libraries.

The Python files import the Python extension as a module, which, in-turn, opens the shared libraries with dlopen.

I am not clear on how to test my application. When building the extension and libraries, they all get built into build/cp311 as described by the documentation. However, when I try to run my application from the Git repo, it fails to import the modules.

I also tried to run the application with PYTHONPATH=build/cp311/ but still it failed to load the modules.

How do I test my application?

Thank you.

You must be logged in to vote

I am not sure what this means. I am using editable builds but do you mean that after the build I would have to copy all of the pure-Python files into the build directory build/cp311?

Editable installs will create a hook loader that ensures that any time you import the project, regardless of current directory and regardless of $PYTHONPATH, you get the version you installed via the editable install. It also has some nice tricks such as automatically reloading any time you modify a source file -- because it will then run ninja to rebuild the C extensions before proceeding with the import, so you can edit source files and rerun your code / tests / interactive interpreter without manually re…

Replies: 1 comment · 4 replies

Comment options

In order for a python project to be importable when it consists of both .py files and compiled extensions, you have to do one of:

  • have them be different top-level importable modules, no shared namespace
  • install the project (at least in editable mode) and merge the source and build trees together

meson-python supports editable installation, so you can install it that way and then iterate on that for your tests. For a more in-depth approach, you can take a look at:

You must be logged in to vote
4 replies
@voidtrance
Comment options

In order for a python project to be importable when it consists of both .py files and compiled extensions, you have to do one of:

  • have them be different top-level importable modules, no shared namespace

They are separate importable modules. However, the pure-Python module cannot find the C extension. This is the source tree after the mason-python build:

.
+- build/
|  +- cp311/
|     +- ...
|     +- src/
|        +- ext/
|           +- myext.cpython-311-x86_64-linux-gnu.so
+- src
|  +- ext
|     +- myext.c
+- module
|  +- __init__.py
+- main.py

main.py imports module needs to import myext. However, module can't find myext even with PYTHONPATH set.

  • install the project (at least in editable mode) and merge the source and build trees together

I am not sure what this means. I am using editable builds but do you mean that after the build I would have to copy all of the pure-Python files into the build directory build/cp311?

@eli-schwartz
Comment options

I am not sure what this means. I am using editable builds but do you mean that after the build I would have to copy all of the pure-Python files into the build directory build/cp311?

Editable installs will create a hook loader that ensures that any time you import the project, regardless of current directory and regardless of $PYTHONPATH, you get the version you installed via the editable install. It also has some nice tricks such as automatically reloading any time you modify a source file -- because it will then run ninja to rebuild the C extensions before proceeding with the import, so you can edit source files and rerun your code / tests / interactive interpreter without manually rebuilding.

Answer selected by voidtrance
@eli-schwartz
Comment options

If PYTHONPATH contains both:

  • $PWD (to execute the statement import module)
  • $PWD/build/cp11/src/ext (to execute the statement import myext)

Then in principle I can't see what the problem would be when running directly from the source tree. Have you tried adding e.g. a debug print to main.py to have it show what it sees as sys.path when the failure occurs?

@orlitzky
Comment options

I think in this example it would work, but if you modify it slightly to:

  • src/mymodule/__init__.py
  • src/mymodule/ext.c

then the issue is that you wind up with two conflicting locations for the same module:

  • src/mymodule (contains the py files)
  • build/src/mymodule (contains the shared libraries)

You can add both to your PYTHONPATH, but python stops searching once it thinks it knows where the module lives. So, for example,

from mymodule.ext import foo

will fail because python sees src/mymodule/__init__.py and decides that "mymodule" lives at src/mymodule.

If I drop the src location from PYTHONPATH, I can import the extension, and if I drop the build location, I can import __init__.py, but getting both to work at the same time is an open problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.