Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 34e82cd

Browse filesBrowse files
philscrickeylev
philsc
and
rickeylev
authored
feat: provide access to arbitrary interpreters (bazel-contrib#2507)
There are some use cases that folks want to cover here. They are discussed in [this Slack thread][1]. The high-level summary is: 1. Users want to run the exact same interpreter that Bazel is running to minimize environmental issues. 2. It is useful to pass a target label to third-party tools like mypy so that they can use the correct interpreter. This patch adds to @rickeylev's work from bazel-contrib#2359 by adding docs and a few integration tests. [1]: https://bazelbuild.slack.com/archives/CA306CEV6/p1730095371089259 --------- Co-authored-by: Richard Levasseur <rlevasseur@google.com>
1 parent 0a3704d commit 34e82cd
Copy full SHA for 34e82cd

13 files changed

+430
-25
lines changed

‎docs/api/rules_python/python/bin/index.md

Copy file name to clipboard
+41Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
:::{default-domain} bzl
2+
:::
3+
:::{bzl:currentfile} //python/bin:BUILD.bazel
4+
:::
5+
6+
# //python/bin
7+
8+
:::{bzl:target} python
9+
10+
A target to directly run a Python interpreter.
11+
12+
By default, it uses the Python version that toolchain resolution matches
13+
(typically the one marked `is_default=True` in `MODULE.bazel`).
14+
15+
This runs a Python interpreter in a similar manner as when running `python3`
16+
on the command line. It can be invoked using `bazel run`. Remember that in
17+
order to pass flags onto the program `--` must be specified to separate
18+
Bazel flags from the program flags.
19+
20+
An example that will run Python 3.12 and have it print the version
21+
22+
```
23+
bazel run @rules_python//python/bin:python \
24+
`--@rule_python//python/config_settings:python_verion=3.12 \
25+
-- \
26+
--version
27+
```
28+
29+
::::{seealso}
30+
The {flag}`--python_src` flag for using the intepreter a binary/test uses.
31+
::::
32+
33+
::::{versionadded} VERSION_NEXT_FEATURE
34+
::::
35+
:::
36+
37+
:::{bzl:flag} python_src
38+
39+
The target (one providing `PyRuntimeInfo`) whose python interpreter to use for
40+
{obj}`:python`.
41+
:::

‎docs/toolchains.md

Copy file name to clipboardExpand all lines: docs/toolchains.md
+43-2Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -396,7 +396,7 @@ provide `Python.h`.
396396

397397
This is typically implemented using {obj}`py_cc_toolchain()`, which provides
398398
{obj}`ToolchainInfo` with the field `py_cc_toolchain` set, which is a
399-
{obj}`PyCcToolchainInfo` provider instance.
399+
{obj}`PyCcToolchainInfo` provider instance.
400400

401401
This toolchain type is intended to hold only _target configuration_ values
402402
relating to the C/C++ information for the Python runtime. As such, when defining
@@ -556,4 +556,45 @@ of available toolchains.
556556
Currently the following flags are used to influence toolchain selection:
557557
* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant.
558558
* {obj}`--@rules_python//python/config_settings:py_freethreaded` for selecting
559-
the freethreaded experimental Python builds available from `3.13.0` onwards.
559+
the freethreaded experimental Python builds available from `3.13.0` onwards.
560+
561+
## Running the underlying interpreter
562+
563+
To run the interpreter that Bazel will use, you can use the
564+
`@rules_python//python/bin:python` target. This is a binary target with
565+
the executable pointing at the `python3` binary plus its relevent runfiles.
566+
567+
```console
568+
$ bazel run @rules_python//python/bin:python
569+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
570+
Type "help", "copyright", "credits" or "license" for more information.
571+
>>>
572+
$ bazel run @rules_python//python/bin:python --@rules_python//python/config_settings:python_version=3.12
573+
Python 3.12.0 (main, Oct 3 2023, 01:27:23) [Clang 17.0.1 ] on linux
574+
Type "help", "copyright", "credits" or "license" for more information.
575+
>>>
576+
```
577+
578+
You can also access a specific binary's interpreter this way by using the
579+
`@rules_python//python/bin:python_src` target. In the example below, it is
580+
assumed that the `@rules_python//tools/publish:twine` binary is fixed at Python
581+
3.11.
582+
583+
```console
584+
$ bazel run @rules_python//python/bin:python --@rules_python//python/bin:interpreter_src=@rules_python//tools/publish:twine
585+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
586+
Type "help", "copyright", "credits" or "license" for more information.
587+
>>>
588+
$ bazel run @rules_python//python/bin:python --@rules_python//python/bin:interpreter_src=@rules_python//tools/publish:twine --@rules_python//python/config_settings:python_version=3.12
589+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
590+
Type "help", "copyright", "credits" or "license" for more information.
591+
>>>
592+
```
593+
Despite setting the Python version explicitly to 3.12 in the example above, the
594+
interpreter comes from the `@rules_python//tools/publish:twine` binary. That is
595+
a fixed version.
596+
597+
:::{note}
598+
The `python` target does not provide access to any modules from `py_*`
599+
targets on its own. Please file a feature request if this is desired.
600+
:::

‎python/BUILD.bazel

Copy file name to clipboardExpand all lines: python/BUILD.bazel
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ filegroup(
3535
name = "distribution",
3636
srcs = glob(["**"]) + [
3737
"//python/api:distribution",
38+
"//python/bin:distribution",
3839
"//python/cc:distribution",
3940
"//python/config_settings:distribution",
4041
"//python/constraints:distribution",

‎python/bin/BUILD.bazel

Copy file name to clipboard
+24Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
load("//python/private:interpreter.bzl", _interpreter_binary = "interpreter_binary")
2+
3+
filegroup(
4+
name = "distribution",
5+
srcs = glob(["**"]),
6+
visibility = ["//:__subpackages__"],
7+
)
8+
9+
_interpreter_binary(
10+
name = "python",
11+
binary = ":python_src",
12+
target_compatible_with = select({
13+
"@platforms//os:windows": ["@platforms//:incompatible"],
14+
"//conditions:default": [],
15+
}),
16+
visibility = ["//visibility:public"],
17+
)
18+
19+
# The user can modify this flag to source different interpreters for the
20+
# `python` target above.
21+
label_flag(
22+
name = "python_src",
23+
build_setting_default = "//python:none",
24+
)

‎python/private/common.bzl

Copy file name to clipboardExpand all lines: python/private/common.bzl
+17Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -543,3 +543,20 @@ def target_platform_has_any_constraint(ctx, constraints):
543543
if ctx.target_platform_has_constraint(constraint_value):
544544
return True
545545
return False
546+
547+
def runfiles_root_path(ctx, short_path):
548+
"""Compute a runfiles-root relative path from `File.short_path`
549+
550+
Args:
551+
ctx: current target ctx
552+
short_path: str, a main-repo relative path from `File.short_path`
553+
554+
Returns:
555+
{type}`str`, a runflies-root relative path
556+
"""
557+
558+
# The ../ comes from short_path is for files in other repos.
559+
if short_path.startswith("../"):
560+
return short_path[3:]
561+
else:
562+
return "{}/{}".format(ctx.workspace_name, short_path)

‎python/private/interpreter.bzl

Copy file name to clipboard
+82Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Copyright 2025 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of the rules to access the underlying Python interpreter."""
16+
17+
load("@bazel_skylib//lib:paths.bzl", "paths")
18+
load("//python:py_runtime_info.bzl", "PyRuntimeInfo")
19+
load(":common.bzl", "runfiles_root_path")
20+
load(":sentinel.bzl", "SentinelInfo")
21+
load(":toolchain_types.bzl", "TARGET_TOOLCHAIN_TYPE")
22+
23+
def _interpreter_binary_impl(ctx):
24+
if SentinelInfo in ctx.attr.binary:
25+
toolchain = ctx.toolchains[TARGET_TOOLCHAIN_TYPE]
26+
runtime = toolchain.py3_runtime
27+
else:
28+
runtime = ctx.attr.binary[PyRuntimeInfo]
29+
30+
# NOTE: We name the output filename after the underlying file name
31+
# because of things like pyenv: they use $0 to determine what to
32+
# re-exec. If it's not a recognized name, then they fail.
33+
if runtime.interpreter:
34+
# In order for this to work both locally and remotely, we create a
35+
# shell script here that re-exec's into the real interpreter. Ideally,
36+
# we'd just use a symlink, but that breaks under certain conditions. If
37+
# we use a ctx.actions.symlink(target=...) then it fails under remote
38+
# execution. If we use ctx.actions.symlink(target_path=...) then it
39+
# behaves differently inside the runfiles tree and outside the runfiles
40+
# tree.
41+
#
42+
# This currently does not work on Windows. Need to find a way to enable
43+
# that.
44+
executable = ctx.actions.declare_file(runtime.interpreter.basename)
45+
ctx.actions.expand_template(
46+
template = ctx.file._template,
47+
output = executable,
48+
substitutions = {
49+
"%target_file%": runfiles_root_path(ctx, runtime.interpreter.short_path),
50+
},
51+
is_executable = True,
52+
)
53+
else:
54+
executable = ctx.actions.declare_symlink(paths.basename(runtime.interpreter_path))
55+
ctx.actions.symlink(output = executable, target_path = runtime.interpreter_path)
56+
57+
return [
58+
DefaultInfo(
59+
executable = executable,
60+
runfiles = ctx.runfiles([executable], transitive_files = runtime.files).merge_all([
61+
ctx.attr._bash_runfiles[DefaultInfo].default_runfiles,
62+
]),
63+
),
64+
]
65+
66+
interpreter_binary = rule(
67+
implementation = _interpreter_binary_impl,
68+
toolchains = [TARGET_TOOLCHAIN_TYPE],
69+
executable = True,
70+
attrs = {
71+
"binary": attr.label(
72+
mandatory = True,
73+
),
74+
"_bash_runfiles": attr.label(
75+
default = "@bazel_tools//tools/bash/runfiles",
76+
),
77+
"_template": attr.label(
78+
default = "//python/private:interpreter_tmpl.sh",
79+
allow_single_file = True,
80+
),
81+
},
82+
)

‎python/private/interpreter_tmpl.sh

Copy file name to clipboard
+23Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
3+
# --- begin runfiles.bash initialization v3 ---
4+
# Copy-pasted from the Bazel Bash runfiles library v3.
5+
set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
6+
# shellcheck disable=SC1090
7+
source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
8+
source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
9+
source "$0.runfiles/$f" 2>/dev/null || \
10+
source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
11+
source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
12+
{ echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
13+
# --- end runfiles.bash initialization v3 ---
14+
15+
set +e # allow us to check for errors more easily
16+
readonly TARGET_FILE="%target_file%"
17+
MAIN_BIN=$(rlocation "$TARGET_FILE")
18+
19+
if [[ -z "$MAIN_BIN" || ! -e "$MAIN_BIN" ]]; then
20+
echo "ERROR: interpreter executable not found: $MAIN_BIN (from $TARGET_FILE)"
21+
exit 1
22+
fi
23+
exec "${MAIN_BIN}" "$@"

‎python/private/py_executable.bzl

Copy file name to clipboardExpand all lines: python/private/py_executable.bzl
+6-22Lines changed: 6 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ load(
4848
"filter_to_py_srcs",
4949
"get_imports",
5050
"is_bool",
51+
"runfiles_root_path",
5152
"target_platform_has_any_constraint",
5253
"union_attrs",
5354
)
@@ -447,7 +448,7 @@ def _create_executable(
447448
)
448449

449450
def _create_zip_main(ctx, *, stage2_bootstrap, runtime_details, venv):
450-
python_binary = _runfiles_root_path(ctx, venv.interpreter.short_path)
451+
python_binary = runfiles_root_path(ctx, venv.interpreter.short_path)
451452
python_binary_actual = venv.interpreter_actual_path
452453

453454
# The location of this file doesn't really matter. It's added to
@@ -522,7 +523,7 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
522523

523524
if not venvs_use_declare_symlink_enabled:
524525
if runtime.interpreter:
525-
interpreter_actual_path = _runfiles_root_path(ctx, runtime.interpreter.short_path)
526+
interpreter_actual_path = runfiles_root_path(ctx, runtime.interpreter.short_path)
526527
else:
527528
interpreter_actual_path = runtime.interpreter_path
528529

@@ -543,11 +544,11 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
543544
# may choose to write what symlink() points to instead.
544545
interpreter = ctx.actions.declare_symlink("{}/bin/{}".format(venv, py_exe_basename))
545546

546-
interpreter_actual_path = _runfiles_root_path(ctx, runtime.interpreter.short_path)
547+
interpreter_actual_path = runfiles_root_path(ctx, runtime.interpreter.short_path)
547548
rel_path = relative_path(
548549
# dirname is necessary because a relative symlink is relative to
549550
# the directory the symlink resides within.
550-
from_ = paths.dirname(_runfiles_root_path(ctx, interpreter.short_path)),
551+
from_ = paths.dirname(runfiles_root_path(ctx, interpreter.short_path)),
551552
to = interpreter_actual_path,
552553
)
553554

@@ -646,23 +647,6 @@ def _create_stage2_bootstrap(
646647
)
647648
return output
648649

649-
def _runfiles_root_path(ctx, short_path):
650-
"""Compute a runfiles-root relative path from `File.short_path`
651-
652-
Args:
653-
ctx: current target ctx
654-
short_path: str, a main-repo relative path from `File.short_path`
655-
656-
Returns:
657-
{type}`str`, a runflies-root relative path
658-
"""
659-
660-
# The ../ comes from short_path is for files in other repos.
661-
if short_path.startswith("../"):
662-
return short_path[3:]
663-
else:
664-
return "{}/{}".format(ctx.workspace_name, short_path)
665-
666650
def _create_stage1_bootstrap(
667651
ctx,
668652
*,
@@ -676,7 +660,7 @@ def _create_stage1_bootstrap(
676660
runtime = runtime_details.effective_runtime
677661

678662
if venv:
679-
python_binary_path = _runfiles_root_path(ctx, venv.interpreter.short_path)
663+
python_binary_path = runfiles_root_path(ctx, venv.interpreter.short_path)
680664
else:
681665
python_binary_path = runtime_details.executable_interpreter_path
682666

‎python/private/site_init_template.py

Copy file name to clipboardExpand all lines: python/private/site_init_template.py
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,9 @@ def _maybe_add_path(path):
163163
if cov_tool:
164164
_print_verbose_coverage(f"Using toolchain coverage_tool {cov_tool}")
165165
elif cov_tool := os.environ.get("PYTHON_COVERAGE"):
166-
_print_verbose_coverage(f"Using env var coverage: PYTHON_COVERAGE={cov_tool}")
166+
_print_verbose_coverage(
167+
f"Using env var coverage: PYTHON_COVERAGE={cov_tool}"
168+
)
167169

168170
if cov_tool:
169171
if os.path.isabs(cov_tool):

‎tests/interpreter/BUILD.bazel

Copy file name to clipboard
+52Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Copyright 2024 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
load(":interpreter_tests.bzl", "PYTHON_VERSIONS_TO_TEST", "py_reconfig_interpreter_tests")
16+
17+
# For this test the interpreter is sourced from the current configuration. That
18+
# means both the interpreter and the test itself are expected to run under the
19+
# same Python version.
20+
py_reconfig_interpreter_tests(
21+
name = "interpreter_version_test",
22+
srcs = ["interpreter_test.py"],
23+
data = [
24+
"//python/bin:python",
25+
],
26+
env = {
27+
"PYTHON_BIN": "$(rootpath //python/bin:python)",
28+
},
29+
main = "interpreter_test.py",
30+
python_versions = PYTHON_VERSIONS_TO_TEST,
31+
)
32+
33+
# For this test the interpreter is sourced from a binary pinned at a specific
34+
# Python version. That means the interpreter and the test itself can run
35+
# different Python versions.
36+
py_reconfig_interpreter_tests(
37+
name = "python_src_test",
38+
srcs = ["interpreter_test.py"],
39+
data = [
40+
"//python/bin:python",
41+
],
42+
env = {
43+
# Since we're grabbing the interpreter from a binary with a fixed
44+
# version, we expect to always see that version. It doesn't matter what
45+
# Python version the test itself is running with.
46+
"EXPECTED_INTERPRETER_VERSION": "3.11",
47+
"PYTHON_BIN": "$(rootpath //python/bin:python)",
48+
},
49+
main = "interpreter_test.py",
50+
python_src = "//tools/publish:twine",
51+
python_versions = PYTHON_VERSIONS_TO_TEST,
52+
)

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.