Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 2e6f8ad

Browse filesBrowse files
rickeylevrickeylev
authored
fix: add flag to use runtime venv creation when using bootstrap=script (bazel-contrib#2590)
The bootstrap=script implementation was changed to use declare_symlink() to create explicit symlinks so its venv works. Unfortunately, this broke packaging rules, which would treat the symlinks as regular files. To fix, introduce a flag that stops using declare_symlink() and instead creates the venv at runtime. Creating a venv at runtime is problematic for various reasons, but this should work well enough until packaging rules are able to handle these raw symlinks. The location of the venv can be somewhat controlled by setting the `RULES_PYTHON_VENVS_ROOT` environment variable. This is to better accommodate cases where using /tmp is problematic. Along the way, sort the environment variable docs by their name. Fixes bazel-contrib#2489
1 parent 33cb431 commit 2e6f8ad
Copy full SHA for 2e6f8ad

File tree

Expand file treeCollapse file tree

14 files changed

+320
-51
lines changed
Filter options
Expand file treeCollapse file tree

14 files changed

+320
-51
lines changed

‎CHANGELOG.md

Copy file name to clipboardExpand all lines: CHANGELOG.md
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,10 @@ Unreleased changes template.
7777
The related issue is [#908](https://github.com/bazelbuild/rules_python/issue/908).
7878
* (sphinxdocs) Do not crash when `tag_class` does not have a populated `doc` value.
7979
Fixes ([#2579](https://github.com/bazelbuild/rules_python/issues/2579)).
80+
* (binaries/tests) Fix packaging when using `--bootstrap_impl=script`: set
81+
{obj}`--venvs_use_declare_symlink=no` to have it not create symlinks at
82+
build time (they will be created at runtime instead).
83+
(Fixes [#2489](https://github.com/bazelbuild/rules_python/issues/2489))
8084

8185
{#v0-0-0-added}
8286
### Added

‎MODULE.bazel

Copy file name to clipboardExpand all lines: MODULE.bazel
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ bazel_dep(name = "rules_testing", version = "0.6.0", dev_dependency = True)
8484
bazel_dep(name = "rules_shell", version = "0.3.0", dev_dependency = True)
8585
bazel_dep(name = "rules_multirun", version = "0.9.0", dev_dependency = True)
8686
bazel_dep(name = "bazel_ci_rules", version = "1.0.0", dev_dependency = True)
87+
bazel_dep(name = "rules_pkg", version = "1.0.1", dev_dependency = True)
8788

8889
# Extra gazelle plugin deps so that WORKSPACE.bzlmod can continue including it for e2e tests.
8990
# We use `WORKSPACE.bzlmod` because it is impossible to have dev-only local overrides.

‎docs/api/rules_python/python/config_settings/index.md

Copy file name to clipboardExpand all lines: docs/api/rules_python/python/config_settings/index.md
+24Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,7 @@ Values:
212212
:::
213213
::::
214214

215+
215216
::::{bzl:flag} bootstrap_impl
216217
Determine how programs implement their startup process.
217218

@@ -258,3 +259,26 @@ Values:
258259
:::
259260

260261
::::
262+
263+
::::{bzl:flag} venvs_use_declare_symlink
264+
265+
Determines if relative symlinks are created using `declare_symlink()` at build
266+
time.
267+
268+
This is only intended to work around
269+
[#2489](https://github.com/bazelbuild/rules_python/issues/2489), where some
270+
packaging rules don't support `declare_symlink()` artifacts.
271+
272+
Values:
273+
* `yes`: Use `declare_symlink()` and create relative symlinks at build time.
274+
* `no`: Do not use `declare_symlink()`. Instead, the venv will be created at
275+
runtime.
276+
277+
:::{seealso}
278+
{envvar}`RULES_PYTHON_EXTRACT_ROOT` for customizing where the runtime venv
279+
is created.
280+
:::
281+
282+
:::{versionadded} VERSION_NEXT_PATCH
283+
:::
284+
::::

‎docs/environment-variables.md

Copy file name to clipboardExpand all lines: docs/environment-variables.md
+56-33Lines changed: 56 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,56 @@
11
# Environment Variables
22

3-
:::{envvar} RULES_PYTHON_REPO_DEBUG
3+
:::{envvar} RULES_PYTHON_BOOTSTRAP_VERBOSE
44

5-
When `1`, repository rules will print debug information about what they're
5+
When `1`, debug information about bootstrapping of a program is printed to
6+
stderr.
7+
:::
8+
9+
:::{envvar} RULES_PYTHON_BZLMOD_DEBUG
10+
11+
When `1`, bzlmod extensions will print debug information about what they're
612
doing. This is mostly useful for development to debug errors.
713
:::
814

9-
:::{envvar} RULES_PYTHON_REPO_DEBUG_VERBOSITY
15+
:::{envvar} RULES_PYTHON_DEPRECATION_WARNINGS
1016

11-
Determines the verbosity of logging output for repo rules. Valid values:
17+
When `1`, the rules_python will warn users about deprecated functionality that will
18+
be removed in a subsequent major `rules_python` version. Defaults to `0` if unset.
19+
:::
1220

13-
* `DEBUG`
14-
* `INFO`
15-
* `TRACE`
21+
:::{envvar} RULES_PYTHON_ENABLE_PYSTAR
22+
23+
When `1`, the rules_python Starlark implementation of the core rules is used
24+
instead of the Bazel-builtin rules. Note this requires Bazel 7+.
1625
:::
1726

18-
:::{envvar} RULES_PYTHON_REPO_TOOLCHAIN_VERSION_OS_ARCH
27+
::::{envvar} RULES_PYTHON_EXTRACT_ROOT
1928

20-
Determines the python interpreter platform to be used for a particular
21-
interpreter `(version, os, arch)` triple to be used in repository rules.
22-
Replace the `VERSION_OS_ARCH` part with actual values when using, e.g.
23-
`3_13_0_linux_x86_64`. The version values must have `_` instead of `.` and the
24-
os, arch values are the same as the ones mentioned in the
25-
`//python:versions.bzl` file.
29+
Directory to use as the root for creating files necessary for bootstrapping so
30+
that a binary can run.
31+
32+
Only applicable when {bzl:flag}`--venvs_use_declare_symlink=no` is used.
33+
34+
When set, a binary will attempt to find a unique, reusable, location within this
35+
directory for the files it needs to create to aid startup. The files may not be
36+
deleted upon program exit; it is the responsibility of the caller to ensure
37+
cleanup.
38+
39+
Manually specifying the directory is useful to lower the overhead of
40+
extracting/creating files on every program execution. By using a location
41+
outside /tmp, longer lived programs don't have to worry about files in /tmp
42+
being cleaned up by the OS.
43+
44+
If not set, then a temporary directory will be created and deleted upon program
45+
exit.
46+
47+
:::{versionadded} VERSION_NEXT_PATCH
48+
:::
49+
::::
50+
51+
:::{envvar} RULES_PYTHON_GAZELLE_VERBOSE
52+
53+
When `1`, debug information from gazelle is printed to stderr.
2654
:::
2755

2856
:::{envvar} RULES_PYTHON_PIP_ISOLATED
@@ -34,37 +62,32 @@ Valid values:
3462
* Other non-empty values mean to use isolated mode.
3563
:::
3664

37-
:::{envvar} RULES_PYTHON_BZLMOD_DEBUG
65+
:::{envvar} RULES_PYTHON_REPO_DEBUG
3866

39-
When `1`, bzlmod extensions will print debug information about what they're
67+
When `1`, repository rules will print debug information about what they're
4068
doing. This is mostly useful for development to debug errors.
4169
:::
4270

43-
:::{envvar} RULES_PYTHON_DEPRECATION_WARNINGS
44-
45-
When `1`, the rules_python will warn users about deprecated functionality that will
46-
be removed in a subsequent major `rules_python` version. Defaults to `0` if unset.
47-
:::
71+
:::{envvar} RULES_PYTHON_REPO_DEBUG_VERBOSITY
4872

49-
:::{envvar} RULES_PYTHON_ENABLE_PYSTAR
73+
Determines the verbosity of logging output for repo rules. Valid values:
5074

51-
When `1`, the rules_python Starlark implementation of the core rules is used
52-
instead of the Bazel-builtin rules. Note this requires Bazel 7+.
75+
* `DEBUG`
76+
* `INFO`
77+
* `TRACE`
5378
:::
5479

55-
:::{envvar} RULES_PYTHON_BOOTSTRAP_VERBOSE
80+
:::{envvar} RULES_PYTHON_REPO_TOOLCHAIN_VERSION_OS_ARCH
5681

57-
When `1`, debug information about bootstrapping of a program is printed to
58-
stderr.
82+
Determines the python interpreter platform to be used for a particular
83+
interpreter `(version, os, arch)` triple to be used in repository rules.
84+
Replace the `VERSION_OS_ARCH` part with actual values when using, e.g.
85+
`3_13_0_linux_x86_64`. The version values must have `_` instead of `.` and the
86+
os, arch values are the same as the ones mentioned in the
87+
`//python:versions.bzl` file.
5988
:::
6089

6190
:::{envvar} VERBOSE_COVERAGE
6291

6392
When `1`, debug information about coverage behavior is printed to stderr.
6493
:::
65-
66-
67-
:::{envvar} RULES_PYTHON_GAZELLE_VERBOSE
68-
69-
When `1`, debug information from gazelle is printed to stderr.
70-
:::

‎python/config_settings/BUILD.bazel

Copy file name to clipboardExpand all lines: python/config_settings/BUILD.bazel
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ load(
99
"LibcFlag",
1010
"PrecompileFlag",
1111
"PrecompileSourceRetentionFlag",
12+
"VenvsUseDeclareSymlinkFlag",
1213
)
1314
load(
1415
"//python/private/pypi:flags.bzl",
@@ -121,6 +122,13 @@ config_setting(
121122
visibility = ["//visibility:public"],
122123
)
123124

125+
string_flag(
126+
name = "venvs_use_declare_symlink",
127+
build_setting_default = VenvsUseDeclareSymlinkFlag.YES,
128+
values = VenvsUseDeclareSymlinkFlag.flag_values(),
129+
visibility = ["//visibility:public"],
130+
)
131+
124132
# pip.parse related flags
125133

126134
string_flag(

‎python/private/flags.bzl

Copy file name to clipboardExpand all lines: python/private/flags.bzl
+15Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,21 @@ PrecompileSourceRetentionFlag = enum(
123123
get_effective_value = _precompile_source_retention_flag_get_effective_value,
124124
)
125125

126+
def _venvs_use_declare_symlink_flag_get_value(ctx):
127+
return ctx.attr._venvs_use_declare_symlink_flag[BuildSettingInfo].value
128+
129+
# Decides if the venv created by bootstrap=script uses declare_file() to
130+
# create relative symlinks. Workaround for #2489 (packaging rules not supporting
131+
# declare_link() files).
132+
# buildifier: disable=name-conventions
133+
VenvsUseDeclareSymlinkFlag = FlagEnum(
134+
# Use declare_file() and relative symlinks in the venv
135+
YES = "yes",
136+
# Do not use declare_file() and relative symlinks in the venv
137+
NO = "no",
138+
get_value = _venvs_use_declare_symlink_flag_get_value,
139+
)
140+
126141
# Used for matching freethreaded toolchains and would have to be used in wheels
127142
# as well.
128143
# buildifier: disable=name-conventions

‎python/private/py_executable.bzl

Copy file name to clipboardExpand all lines: python/private/py_executable.bzl
+27-6Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ load(
5151
"target_platform_has_any_constraint",
5252
"union_attrs",
5353
)
54-
load(":flags.bzl", "BootstrapImplFlag")
54+
load(":flags.bzl", "BootstrapImplFlag", "VenvsUseDeclareSymlinkFlag")
5555
load(":precompile.bzl", "maybe_precompile")
5656
load(":py_cc_link_params_info.bzl", "PyCcLinkParamsInfo")
5757
load(":py_executable_info.bzl", "PyExecutableInfo")
@@ -195,6 +195,10 @@ accepting arbitrary Python versions.
195195
"_python_version_flag": attr.label(
196196
default = "//python/config_settings:python_version",
197197
),
198+
"_venvs_use_declare_symlink_flag": attr.label(
199+
default = "//python/config_settings:venvs_use_declare_symlink",
200+
providers = [BuildSettingInfo],
201+
),
198202
"_windows_constraints": attr.label_list(
199203
default = [
200204
"@platforms//os:windows",
@@ -512,7 +516,25 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
512516
ctx.actions.write(pyvenv_cfg, "")
513517

514518
runtime = runtime_details.effective_runtime
515-
if runtime.interpreter:
519+
venvs_use_declare_symlink_enabled = (
520+
VenvsUseDeclareSymlinkFlag.get_value(ctx) == VenvsUseDeclareSymlinkFlag.YES
521+
)
522+
523+
if not venvs_use_declare_symlink_enabled:
524+
if runtime.interpreter:
525+
interpreter_actual_path = _runfiles_root_path(ctx, runtime.interpreter.short_path)
526+
else:
527+
interpreter_actual_path = runtime.interpreter_path
528+
529+
py_exe_basename = paths.basename(interpreter_actual_path)
530+
531+
# When the venv symlinks are disabled, the $venv/bin/python3 file isn't
532+
# needed or used at runtime. However, the zip code uses the interpreter
533+
# File object to figure out some paths.
534+
interpreter = ctx.actions.declare_file("{}/bin/{}".format(venv, py_exe_basename))
535+
ctx.actions.write(interpreter, "actual:{}".format(interpreter_actual_path))
536+
537+
elif runtime.interpreter:
516538
py_exe_basename = paths.basename(runtime.interpreter.short_path)
517539

518540
# Even though ctx.actions.symlink() is used, using
@@ -571,6 +593,7 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
571593

572594
return struct(
573595
interpreter = interpreter,
596+
recreate_venv_at_runtime = not venvs_use_declare_symlink_enabled,
574597
# Runfiles root relative path or absolute path
575598
interpreter_actual_path = interpreter_actual_path,
576599
files_without_interpreter = [pyvenv_cfg, pth, site_init],
@@ -657,15 +680,13 @@ def _create_stage1_bootstrap(
657680
else:
658681
python_binary_path = runtime_details.executable_interpreter_path
659682

660-
if is_for_zip and venv:
661-
python_binary_actual = venv.interpreter_actual_path
662-
else:
663-
python_binary_actual = ""
683+
python_binary_actual = venv.interpreter_actual_path if venv else ""
664684

665685
subs = {
666686
"%is_zipfile%": "1" if is_for_zip else "0",
667687
"%python_binary%": python_binary_path,
668688
"%python_binary_actual%": python_binary_actual,
689+
"%recreate_venv_at_runtime%": str(int(venv.recreate_venv_at_runtime)) if venv else "0",
669690
"%target%": str(ctx.label),
670691
"%workspace_name%": ctx.workspace_name,
671692
}

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.