Skip to content

Navigation Menu

Sign in

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

pip.parse vendored support #2508

Unanswered
gfrankliu asked this question in Q&A
pip.parse vendored support #2508
@gfrankliu gfrankliu
Dec 2, 2024 · 4 comments · 2 replies
Return to top
Discussion options

gfrankliu
Dec 2, 2024

We are currently using WORKSPACE and pip_parse_vendored following the examples in https://github.com/bazelbuild/rules_python/tree/main/examples/pip_parse_vendored

While migrating to MODULE.bazel, there are no similar pip.parse_vendored example. In particular, I am looking for the equivalent of https://github.com/bazelbuild/rules_python/blob/main/examples/pip_parse_vendored/BUILD.bazel#L24-L34

Looking at pip_parse generated @pip_deps_to_be_vendored//:requirements.bzl and pip.parse generated :requirements.bzl , I see the latter misses _config = { ... } and _packages = [ ... ]
Is that a bug?
You must be logged in to vote

Replies: 4 comments · 2 replies

Comment options

gfrankliu
Dec 2, 2024
Author

Here is a quick test example to show the differences in the requirements.bzl

Using WORKSPACE and pip_parse:

# WORKSPACE
python_register_toolchains(
    name = "python_interpreter",
    ignore_root_user_error = True,
    python_version = "3.10.11",
)
load("@python_interpreter//:defs.bzl", "interpreter")
pip_parse(
    name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

Using the new MODULE.bazel and pip.parse:

# MODULE.bazel
python.toolchain(
    ignore_root_user_error = True,
    is_default = True,
    python_version = "3.10.11",
)
use_repo(python, "python_3_10_11")
pip.parse(
    hub_name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_version = "3.10.11",
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

You can see the new test.bzl misses:

load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")

_packages =
_config = 
def entry_point(pkg, script = None):
def _get_annotation(requirement):
def install_deps(**whl_library_kwargs):

Is this a bug in the new pip.parse?
You must be logged in to vote
0 replies
Comment options

aignas
Dec 3, 2024
Maintainer

bzlmod has lockfile mechanism and its own vendoring code. Does it not satisfy your needs?

There are no plans to add vendoring in the same way it was done in WORKSPACE.
You must be logged in to vote
0 replies
Comment options

gfrankliu
Dec 15, 2024
Author

We have some internal custom bazel rules that

  • uses _config (specifically extra_pip_args from _config) from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses the whole _config.
  • use _packages from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses _packages

The python modules directory under runfiles also look different, eg: module httplib2 looks like app.runfiles/third_party_lib_httplib2/site-packages/ when using WORKSPACE/pip_parse, but looks like app.runfiles/rules_python~~pip~third_party_lib_310_httplib2/site-packages/ when using bzmod/pip.parse

Is it possible to configure pip.parse to behave more like pip_parse to make the rules_python WORKSPACE to bzlmod migration easier?
You must be logged in to vote
0 replies
Comment options

aignas
Dec 16, 2024
Maintainer

It seems that you may be depending on implementation details within rules_python and unfortunately some things just are not possible in bzlmod:

  • Having extra_pip_args might be possible, but in general they can be different for each target platform, so supporting that might be tough. What is the use case here?
  • What is the use case for _packages?
  • The layout is not controlled by us and is imposed by bzlmod itself, so unfortunately that won't be possible.

You can go the other way though to first make pip_parse closer to what pip.parse is. We tried to make it possible to do everything that you can do through the API exposed by pip_parse (i.e. the attributes and the publicly available constants in requirements.bzl) to be possible in pip.parse bzlmod extension.

Since there is nothing to do here, I will migrate this to a discussion.
You must be logged in to vote
2 replies
@gfrankliu
Comment options

gfrankliu Dec 16, 2024
Author

  • I see in my pip_parse generated requirements.bzl, extra_pip_args came from the requirements.in. eg: my requirements.in has
--extra-index-url https://download.pytorch.org/whl/cu117
-f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

and my requirements.bzl has corresponding:

    "extra_pip_args": ["--extra-index-url", "https://download.pytorch.org/whl/cu117", "--find-links", "https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"],

Does that mean supporting this in the pip.parsed requirement.bzl should be a straightforward copy from requirements.in?
@aignas
Comment options

aignas Dec 17, 2024
Maintainer

Does that mean supporting this in the pip.parsed requirement.bzl should be a straightforward copy from requirements.in?

I don't think it makes sense adding it in there as it may complicate things.

Those custom bazel rules we have were written by someone who's no longer on the team. I am still trying to understand. It seems to get extra_pip_args and _packages to inject into the Dataflow Apache beam python so that Dataflow cloud runner can use pip behind the scene as part of the Dataflow job startup:

You could have a separate build rule that is duplicating the logic on your end. If the data is in the requirements.in file already, you can read the same data yourself. To be honest, I see no reason for rules_python to be involved in here. However, I understand the pain of maintaining/fixing code that you did not write, so I wish you the best of luck with that. :)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
@gfrankliu @aignas
Converted from issue

This discussion was converted from issue #2462 on December 16, 2024 03:13.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.