Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

WORKSPACE to MODULE.bazel migration setting default python version #2509

Answered by aignas
gfrankliu asked this question in Q&A
WORKSPACE to MODULE.bazel migration setting default python version #2509
@gfrankliu gfrankliu
Dec 14, 2024 · 4 comments · 5 replies
Answered by aignas Return to top
Discussion options

gfrankliu
Dec 14, 2024

We are migrating from WORKSPACE to MODULE.bazel. Most of modules are now in MODULE.bazel and very few are still in WORKSPACE.

We have rules_python still in WORKSPACE because some of our internal custom python codes/rules/builds still need more time to update.

We use python version 3.10, eg: in WORKSPACE:

python_register_toolchains(
    name = "python_interpreter",
    python_version = "3.10",
)

We notice an issue where our external dependencies from MODULE.bazel are using python 3.11 instead of our default 3.10.

Since we are still using WORKSPACE and can't set python.toolchain(is_default = True, python_version = "3.10") in our root MODULE.bazel, how can we "tell" those external modules (they are already using MODULE.bazel) to use python 3.10?
You must be logged in to vote
Answered by aignas Dec 16, 2024

python_3_10 is something that points to an interpreter that can only be used in the build context. python_3_10_host uses the same repositories but resolves/copies the files in the repository_rule context. The legacy load("@python_3_10_host//:defs.bzl", "interpreter") has been replaced with @python_3_10_host//:python.

View full answer

Replies: 4 comments · 5 replies

Comment options

gfrankliu
Dec 15, 2024
Author

We are using rules_python 0.40.0, and latest bazel 7: bazel 7.4.1
You must be logged in to vote
0 replies
Comment options

aignas
Dec 15, 2024
Maintainer

That might be working as intended - MODULE.bazel dependencies are using the default toolchain set by bzlmod code and it cannot know anything about the WORKSPACE paths.

I am not sure if this is something that can be easily fixed.

At the same time, I am not sure why you can define your toolchain in bzlmod? You should still be able to use pip_parse from WORKSPACE with the toolchain set in MODULE.bazel.
You must be logged in to vote
0 replies
Comment options

gfrankliu
Dec 15, 2024
Author

Thanks for the suggestion! I just tried this:

In MODULE.bazel, I added:

bazel_dep(name = "rules_python", version = "0.40.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(is_default = True, python_version = "3.10")
use_repo(python, "python_3_10")

In WORKSPACE, I only have below lines, nothing else:

load("@rules_python//python:pip.bzl", "pip_parse")
pip_parse(
    name = "third_party_lib",
    requirements_lock = "//:requirements.txt",
)

bazel build gives this error:

ERROR: Failed to load Starlark extension '@@rules_python_internal//:rules_python_config.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @@rules_python_internal
This could either mean you have to add the '@@rules_python_internal' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
ERROR: Error computing the main repository mapping: cycles detected during computation of main repo mapping

Since pip.parse isn't 100% matching pip_parse, it breaks some of our internal custom bazel rules. It will take some time for us to update all those custom rules. In the meantime, we would like to migrate non python stuff into MODULE.bazel.
You must be logged in to vote
1 reply
@aignas
Comment options

aignas Dec 16, 2024
Maintainer

You need the rules_python deps in the WORKSPACE as well as the error message might be hinting you.

With rules_python 1.0 and 7.x bazel, the following WORKSPACE file works:

$ cat WORKSPACE
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "third_party",
    requirements_lock = "//:requirements.txt",
)
Comment options

gfrankliu
Dec 16, 2024
Author

Thanks and that gets me further, but it fails when I pin the version to python 3.10 in WORKSPACE

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")
load("@python_3_10//:defs.bzl", "interpreter")

pip_parse(
    name = "third_party",
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

load("@third_party//:requirements.bzl", "install_deps")

install_deps()

bazel build gives errors:

	File "/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/b71142cf8050563176d73cc4d125110e/external/rules_python~/python/private/pypi/whl_library.bzl", line 178, column 68, in _whl_library_impl
		python_interpreter = pypi_repo_utils.resolve_python_interpreter(
	File "/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/b71142cf8050563176d73cc4d125110e/external/rules_python~/python/private/pypi/pypi_repo_utils.bzl", line 67, column 40, in _resolve_python_interpreter
		python_interpreter = mrctx.path(root_build_bazel).dirname.get_child(python_interpreter_target.name)
Error in path: Unable to load package for @@python_3_10_x86_64-unknown-linux-gnu//:BUILD.bazel: The repository '@@python_3_10_x86_64-unknown-linux-gnu' could not be resolved: Repository '@@python_3_10_x86_64-unknown-linux-gnu' is not defined

In the requirement.bzl _config section, I can see

    "python_interpreter": "python3",
    "python_interpreter_target": "@@python_3_10_x86_64-unknown-linux-gnu//:bin/python3",

The bazel errors says the repo @@python_3_10_x86_64-unknown-linux-gnu doesn't exist. In the bazel cache directory, I do see "external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu" but don't see "external/python_3_10_x86_64-unknown-linux-gnu".

I was using rules_python 0.38 in this test. If I switch to 1.0.0, I get a different error:

		load("@python_3_10//:defs.bzl", "interpreter")
Error: file '@python_3_10//:defs.bzl' does not contain symbol 'interpreter'
You must be logged in to vote
4 replies
@aignas
Comment options

aignas Dec 16, 2024
Maintainer

This works for me:

$ cat WORKSPACE
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "third_party",
    requirements_lock = "//:requirements.txt",
    python_interpreter_target = "@python_3_10_host//:python",
)

load("@third_party//:requirements.bzl", "install_deps")

install_deps()

$ cat MODULE.bazel
bazel_dep(name = "rules_python", version = "0.40.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")

python.toolchain(is_default = True, python_version = "3.10")

use_repo(python, "python_3_10", "python_3_10_host")
@aignas
Comment options

aignas Dec 16, 2024
Maintainer

I have added a PR to add this snippet to our docs as this may be also useful to others: #2510
@gfrankliu
Comment options

gfrankliu Dec 16, 2024
Author

Thanks @aignas
I was using @python_3_10 and I can see you are using @python_3_10_host in your example. Have you tried @python_3_10 ?

With your hint, I just tried @python_3_10_host , which seems to work. Just curious why @python_3_10 doesn't work.
@aignas
Comment options

aignas Dec 16, 2024
Maintainer

python_3_10 is something that points to an interpreter that can only be used in the build context. python_3_10_host uses the same repositories but resolves/copies the files in the repository_rule context. The legacy load("@python_3_10_host//:defs.bzl", "interpreter") has been replaced with @python_3_10_host//:python.
Answer selected by gfrankliu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
@gfrankliu @aignas
Converted from issue

This discussion was converted from issue #2505 on December 16, 2024 03:25.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.