Awesome thanks as always for the input @rgommers

Working through this some more, I think I actually got to a good place doing:

arrow-nanoarrow/
  meson.build
  python/
    meson.build
    subproject/
      arrow-nanoarrow-symlink/  -- symlink to arrow-nanoarrow project root
      arrow-nanoarrow.wrap  -- in theory only used when symlink is not present

Ideally I was hoping that the subproject structure would have just been:

subprojects/
    arrow-nanoarrow/
    arrow-nanoarrow.wrap

And the wrap file would only come into play when the symlink is not resolvable (i.e. when we package an sdist, that symlink will not be included). However, it does appear that there is an upstream "bug" for this in Meson, so the naming distinction between the symlink and the wrapfile seems to be working for the time being:

mesonbuild/meson#13746

Checking back in on this, are things working for you with your wrap file solution @WillAyd?

An alternative is to put the pyproject.toml one level higher up

This should be the recommended way of doing things for projects where that's an option I believe (the discussion thread in the next answer below doesn't change that so far). It should avoid any problems with subprojects, sdist generation, etc.

If for policy reasons in a very large repo the pyproject.toml isn't allowed to live one level up from the python/ bindings sub-directory, then the symlink + wrap file approach seems like a decent alternative.

Haven't finished it but yes so far the symlink + wrap is workable. Although in our case we've replaced the wrap file with a dist hook script as you suggested, which works just as well

Many projects choose to have a python/ subdirectory in the same git repository for Python bindings that are tightly coupled to the underlying library (e.g., arrow/pyarrow, duckdb, protobuf). For Apache projects there is also a complication that a "release" requires quite a lot of overhead + a vote, and creating an extra repository requires intervention from the ASF (it does happen though, for example apache/datafusion-python). This is particularly true when starting Python bindings (i.e., at the beginning many coupled changes are required, but as the underlying library matures it becomes possible to release the bindings independently). We are definitely still in the "starting Python bindings" phase in nanoarrow 🙂 .

I'm even more confused about what you are trying to achieve. From my understanding of the explanations provided so far, you have two conflicting goals: decouple the Python bindings from the underlying library (this is the only reason that I can see to have them distributed in separate source release archives) but also keep them tightly coupled (because of the code inter-dependencies and unsettled API).

My understanding is that building the Python wrappers requires a local build of the underlying library. If this is the case, I don't understand why you want to distribute the Python wrappers source code and the library source code as separate source archives. The only reason I can envision for doing so would be to allow to build against a locally installed nanoarrow library. However, this does not seem to be supported nor really possible at this stage of development. Is this the case?

Particularly for the nanoarrow library, where some changes in Python may depend on changes being made in the C libraries as well, wouldn't splitting it into two separate repositories make things harder? I'm thinking through a scenario where you have to change the C structure, commit it, then go to another repository for Python, make changes there, only to find that you missed something back in the C repository, etc...

I don't think so. You just work on the checkout of nanoarrow that happens to be in the subprojects folder of the Python bindings project. You can include nanoarrow in the the subprojects directory of the Python bindings as a git submodule. However, I understand that separating the repositories is not something you want to do.

If separating the repositories is not something you want or can do, I don't think the subproject model adapts very well. I would reorganize things so that the Python bindings are built as part of the the build system of the nanoarrow library. If your build system of choice is CMake, you can build Python wheels with scikit-build.

I am not sure I've done a good job communicating the motivation here, although I am not personally offended if this is not a use case meson-python would like to handle (setuptools is not currently causing any problems). Some of the projects I mentioned choose to distribute a self-contained sdist (e.g., duckdb) and some of them do not. I believe this is to accommodate users that need to be able to install in an airgapped environment.

meson-python has nothing to do with the difficulties you are encountering. meson-python is just a way to invoke meson and package the built artifacts. You are trying to bend the Meson subprojects system to do things it is not designed for. I'm not saying that it cannot be made to work but bending it in this way seems much more work that using the build system as intended.

This is also what I meant in my first comment with "put the pyproject.toml one level higher up".