Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit b92927d

Browse filesBrowse files
aignasrickeylev
aignas
and
rickeylev
authored
feat(bzlmod): add python.override APIs (#2222)
Before this PR the users could not override how the hermetic toolchain is downloaded when in `bzlmod`. However, the same APIs would be available to users using `WORKSPACE`. With this we also allow root modules to restrict which toolchain versions the non-root modules, which may be helpful when optimizing the CI runtimes so that we don't waste time downloading multiple `micro` versions of the same `3.X` python version, which most of the times have identical behavior. Whilst at it, tweak the `semver` implementation to allow for testing of absence of values in the original input. Work towards #2081 and this should be one of the last items that are blocking #1361 from the API point of view. Replaces #2151. --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com>
1 parent 93eda70 commit b92927d
Copy full SHA for b92927d

File tree

Expand file treeCollapse file tree

19 files changed

+1137
-142
lines changed
Filter options
Expand file treeCollapse file tree

19 files changed

+1137
-142
lines changed

‎.bazelrc

Copy file name to clipboardExpand all lines: .bazelrc
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, execute
66
# `bazel run @rules_bazel_integration_test//tools:update_deleted_packages`
7-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/python/private,gazelle/pythonconfig,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered
8-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/python/private,gazelle/pythonconfig,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered
7+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered
8+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patches,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,gazelle/python/private,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/custom_commands,tests/integration/ignore_root_user_error,tests/integration/ignore_root_user_error/submodule,tests/integration/local_toolchains,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/py_cc_toolchain_registered
99

1010
test --test_output=errors
1111

‎CHANGELOG.md

Copy file name to clipboardExpand all lines: CHANGELOG.md
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ A brief description of the categories of changes:
2828
* (gazelle): Update error messages when unable to resolve a dependency to be more human-friendly.
2929
* (flags) The {obj}`--python_version` flag now also returns
3030
{obj}`config_common.FeatureFlagInfo`.
31+
* (toolchain): The toolchain patches now expose the `patch_strip` attribute
32+
that one should use when patching toolchains. Please set it if you are
33+
patching python interpreter. In the next release the default will be set to
34+
`0` which better reflects the defaults used in public `bazel` APIs.
3135
* (toolchains) When {obj}`py_runtime.interpreter_version_info` isn't specified,
3236
the {obj}`--python_version` flag will determine the value. This allows
3337
specifying the build-time Python version for the
@@ -65,6 +69,10 @@ A brief description of the categories of changes:
6569

6670

6771
### Added
72+
* (bzlmod): Toolchain overrides can now be done using the new
73+
{bzl:obj}`python.override`, {bzl:obj}`python.single_version_override` and
74+
{bzl:obj}`python.single_version_platform_override` tag classes.
75+
See [#2081](https://github.com/bazelbuild/rules_python/issues/2081).
6876
* (rules) Executables provide {obj}`PyExecutableInfo`, which contains
6977
executable-specific information useful for packaging an executable or
7078
or deriving a new one from the original.

‎MODULE.bazel

Copy file name to clipboardExpand all lines: MODULE.bazel
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ dev_python = use_extension(
8181
"python",
8282
dev_dependency = True,
8383
)
84-
dev_python.rules_python_private_testing(
84+
dev_python.override(
8585
register_all_versions = True,
8686
)
8787

‎docs/toolchains.md

Copy file name to clipboardExpand all lines: docs/toolchains.md
+16-1Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,21 @@ Variables](https://bazel.build/reference/be/make-variables). See the
169169
{gh-path}`test_current_py_toolchain <tests/load_from_macro/BUILD.bazel>` target
170170
for an example.
171171

172+
### Overriding toolchain defaults and adding more versions
173+
174+
One can perform various overrides for the registered toolchains from the root
175+
module. For example, the following use cases would be supported using the
176+
existing attributes:
177+
178+
* Limiting the available toolchains for the entire `bzlmod` transitive graph
179+
via {attr}`python.override.available_python_versions`.
180+
* Setting particular `X.Y.Z` Python versions when modules request `X.Y` version
181+
via {attr}`python.override.minor_mapping`.
182+
* Per-version control of the coverage tool used using
183+
{attr}`python.single_version_platform_override.coverage_tool`.
184+
* Adding additional Python versions via {bzl:obj}`python.single_version_override` or
185+
{bzl:obj}`python.single_version_platform_override`.
186+
172187
## Workspace configuration
173188

174189
To import rules_python in your project, you first need to add it to your
@@ -244,7 +259,7 @@ automatically registers a higher-priority toolchain; it won't be used unless
244259
there is a toolchain misconfiguration somewhere.
245260

246261
To aid migration off the Bazel-builtin toolchain, rules_python provides
247-
{obj}`@rules_python//python/runtime_env_toolchains:all`. This is an equivalent
262+
{bzl:obj}`@rules_python//python/runtime_env_toolchains:all`. This is an equivalent
248263
toolchain, but is implemented using rules_python's objects.
249264

250265

‎examples/bzlmod/MODULE.bazel

Copy file name to clipboardExpand all lines: examples/bzlmod/MODULE.bazel
+50-1Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ bazel_dep(name = "protobuf", version = "24.4", repo_name = "com_google_protobuf"
2222
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
2323
python.toolchain(
2424
configure_coverage_tool = True,
25-
# Only set when you have mulitple toolchain versions.
25+
# Only set when you have multiple toolchain versions.
2626
is_default = True,
2727
python_version = "3.9",
2828
)
@@ -37,6 +37,55 @@ python.toolchain(
3737
python_version = "3.10",
3838
)
3939

40+
# One can override the actual toolchain versions that are available, which can be useful
41+
# when optimizing what gets downloaded and when.
42+
python.override(
43+
available_python_versions = [
44+
"3.10.9",
45+
"3.9.19",
46+
# The following is used by the `other_module` and we need to include it here
47+
# as well.
48+
"3.11.8",
49+
],
50+
# Also override the `minor_mapping` so that the root module,
51+
# instead of rules_python's defaults, controls what full version
52+
# is used when `3.x` is requested.
53+
minor_mapping = {
54+
"3.10": "3.10.9",
55+
"3.11": "3.11.8",
56+
"3.9": "3.9.19",
57+
},
58+
)
59+
60+
# Or the sources that the toolchains come from for all platforms
61+
python.single_version_override(
62+
patch_strip = 1,
63+
# The user can specify patches to be applied to all interpreters.
64+
patches = [],
65+
python_version = "3.10.2",
66+
sha256 = {
67+
"aarch64-apple-darwin": "1409acd9a506e2d1d3b65c1488db4e40d8f19d09a7df099667c87a506f71c0ef",
68+
"aarch64-unknown-linux-gnu": "8f351a8cc348bb45c0f95b8634c8345ec6e749e483384188ad865b7428342703",
69+
"x86_64-apple-darwin": "8146ad4390710ec69b316a5649912df0247d35f4a42e2aa9615bffd87b3e235a",
70+
"x86_64-pc-windows-msvc": "a1d9a594cd3103baa24937ad9150c1a389544b4350e859200b3e5c036ac352bd",
71+
"x86_64-unknown-linux-gnu": "9b64eca2a94f7aff9409ad70bdaa7fbbf8148692662e764401883957943620dd",
72+
},
73+
urls = ["20220227/cpython-{python_version}+20220227-{platform}-{build}.tar.gz"],
74+
)
75+
76+
# Or a single platform. This can be used in combination with the
77+
# `single_version_override` and `single_version_platform_override` will be
78+
# applied after `single_version_override`. Any values present in this override
79+
# will overwrite the values set by the `single_version_override`
80+
python.single_version_platform_override(
81+
patch_strip = 1,
82+
patches = [],
83+
platform = "aarch64-apple-darwin",
84+
python_version = "3.10.2",
85+
sha256 = "1409acd9a506e2d1d3b65c1488db4e40d8f19d09a7df099667c87a506f71c0ef",
86+
urls = ["20220227/cpython-{python_version}+20220227-{platform}-{build}.tar.gz"],
87+
)
88+
4089
# You only need to load this repositories if you are using multiple Python versions.
4190
# See the tests folder for various examples on using multiple Python versions.
4291
# The names "python_3_9" and "python_3_10" are autmatically created by the repo

‎examples/bzlmod/MODULE.bazel.lock

Copy file name to clipboardExpand all lines: examples/bzlmod/MODULE.bazel.lock
+2-2Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎python/extensions/python.bzl

Copy file name to clipboardExpand all lines: python/extensions/python.bzl
+32-1Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,38 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
"Python toolchain module extensions for use with bzlmod"
15+
"""Python toolchain module extensions for use with bzlmod.
16+
17+
:::{topic} Basic usage
18+
19+
The simplest way to configure the toolchain with `rules_python` is as follows.
20+
21+
```starlark
22+
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
23+
python.toolchain(
24+
is_default = True,
25+
python_version = "3.11",
26+
)
27+
use_repo(python, "python_3_11")
28+
```
29+
30+
::::{seealso}
31+
For more in-depth documentation see the {obj}`python.toolchain`.
32+
::::
33+
:::
34+
35+
:::{topic} Overrides
36+
37+
Overrides can be done at 3 different levels:
38+
* Overrides affecting all python toolchain versions on all platforms - {obj}`python.override`.
39+
* Overrides affecting a single toolchain versions on all platforms - {obj}`python.single_version_override`.
40+
* Overrides affecting a single toolchain versions on a single platforms - {obj}`python.single_version_platform_override`.
41+
42+
::::{seealso}
43+
The main documentation page on registering [toolchains](/toolchains).
44+
::::
45+
:::
46+
"""
1647

1748
load("//python/private:python.bzl", _python = "python")
1849

‎python/private/BUILD.bazel

Copy file name to clipboardExpand all lines: python/private/BUILD.bazel
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,11 +141,12 @@ bzl_library(
141141
srcs = ["python.bzl"],
142142
deps = [
143143
":full_version_bzl",
144+
":python_register_toolchains_bzl",
144145
":pythons_hub_bzl",
145146
":repo_utils_bzl",
147+
":semver_bzl",
146148
":toolchains_repo_bzl",
147149
":util_bzl",
148-
"//python:repositories_bzl",
149150
"@bazel_features//:features",
150151
],
151152
)
@@ -202,7 +203,6 @@ bzl_library(
202203
name = "pythons_hub_bzl",
203204
srcs = ["pythons_hub.bzl"],
204205
deps = [
205-
":full_version_bzl",
206206
":py_toolchain_suite_bzl",
207207
],
208208
)

‎python/private/common/py_runtime_rule.bzl

Copy file name to clipboardExpand all lines: python/private/common/py_runtime_rule.bzl
+4-4Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -202,8 +202,8 @@ See @bazel_tools//tools/python:python_bootstrap_template.txt for more variables.
202202
"coverage_tool": attr.label(
203203
allow_files = False,
204204
doc = """
205-
This is a target to use for collecting code coverage information from `py_binary`
206-
and `py_test` targets.
205+
This is a target to use for collecting code coverage information from
206+
{rule}`py_binary` and {rule}`py_test` targets.
207207
208208
If set, the target must either produce a single file or be an executable target.
209209
The path to the single file, or the executable if the target is executable,
@@ -212,7 +212,7 @@ runfiles will be added to the runfiles when coverage is enabled.
212212
213213
The entry point for the tool must be loadable by a Python interpreter (e.g. a
214214
`.py` or `.pyc` file). It must accept the command line arguments
215-
of coverage.py (https://coverage.readthedocs.io), at least including
215+
of [`coverage.py`](https://coverage.readthedocs.io), at least including
216216
the `run` and `lcov` subcommands.
217217
""",
218218
),
@@ -311,7 +311,7 @@ The template to use when two stage bootstrapping is enabled
311311
default = DEFAULT_STUB_SHEBANG,
312312
doc = """
313313
"Shebang" expression prepended to the bootstrapping Python stub script
314-
used when executing `py_binary` targets.
314+
used when executing {rule}`py_binary` targets.
315315
316316
See https://github.com/bazelbuild/bazel/issues/8685 for
317317
motivation.

‎python/private/py_repositories.bzl

Copy file name to clipboardExpand all lines: python/private/py_repositories.bzl
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,8 @@ def http_archive(**kwargs):
2525
def py_repositories():
2626
"""Runtime dependencies that users must install.
2727
28-
This function should be loaded and called in the user's WORKSPACE.
29-
With bzlmod enabled, this function is not needed since MODULE.bazel handles transitive deps.
28+
This function should be loaded and called in the user's `WORKSPACE`.
29+
With `bzlmod` enabled, this function is not needed since `MODULE.bazel` handles transitive deps.
3030
"""
3131
maybe(
3232
internal_config_repo,

‎python/private/pypi/BUILD.bazel

Copy file name to clipboardExpand all lines: python/private/pypi/BUILD.bazel
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,6 @@ bzl_library(
5959
srcs = ["extension.bzl"],
6060
deps = [
6161
":attrs_bzl",
62-
"//python/private:semver_bzl",
6362
":hub_repository_bzl",
6463
":parse_requirements_bzl",
6564
":evaluate_markers_bzl",
@@ -71,6 +70,7 @@ bzl_library(
7170
"//python/private:full_version_bzl",
7271
"//python/private:normalize_name_bzl",
7372
"//python/private:version_label_bzl",
73+
"//python/private:semver_bzl",
7474
"@bazel_features//:features",
7575
] + [
7676
"@pythons_hub//:interpreters_bzl",

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.