Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit a4b946b

Browse filesBrowse files
aignasrickeylev
aignas
and
rickeylev
authored
feat: add an env variable to toggle pipstar (#2855)
This is a flag to start leveraging of the new code paths. The Starlark implementation has been added in 1.4 and has been reverted in the latest release candidates. The `env` variable will be a good way to roll it out more gradually and get more testing. For now we are switching only the `whl_library` internals as the `requirements.txt` files from `uv` may use `*` in `python_full_version` and `platform_version` that are not yet fully supported (#2826). Main goals for this is to start using Starlark implementation so that we don't have any hidden variables. What is more, having this in Starlark is the most maintainable long-term solution for supporting cross-platform builds. Work towards #260 --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com>
1 parent 4ccf5b2 commit a4b946b
Copy full SHA for a4b946b

File tree

Expand file treeCollapse file tree

7 files changed

+187
-86
lines changed
Filter options
Expand file treeCollapse file tree

7 files changed

+187
-86
lines changed

‎CHANGELOG.md

Copy file name to clipboardExpand all lines: CHANGELOG.md
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,9 @@ END_UNRELEASED_TEMPLATE
8686
(the default), the subprocess's stdout/stderr will be logged.
8787
* (toolchains) Local toolchains can be activated with custom flags. See
8888
[Conditionally using local toolchains] docs for how to configure.
89+
* (pypi) `RULES_PYTHON_ENABLE_PIPSTAR` environment variable: when `1`, the Starlark
90+
implementation of wheel METADATA parsing is used (which has improved multi-platform
91+
build support).
8992

9093
{#v0-0-0-removed}
9194
### Removed

‎docs/environment-variables.md

Copy file name to clipboardExpand all lines: docs/environment-variables.md
+9Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,15 @@ The default became `1` if unspecified
6060
:::
6161
::::
6262

63+
::::{envvar} RULES_PYTHON_ENABLE_PIPSTAR
64+
65+
When `1`, the rules_python Starlark implementation of the pypi/pip integration is used
66+
instead of the legacy Python scripts.
67+
68+
:::{versionadded} VERSION_NEXT_FEATURE
69+
:::
70+
::::
71+
6372
::::{envvar} RULES_PYTHON_EXTRACT_ROOT
6473

6574
Directory to use as the root for creating files necessary for bootstrapping so

‎python/private/internal_config_repo.bzl

Copy file name to clipboardExpand all lines: python/private/internal_config_repo.bzl
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ settings for rules to later use.
2020

2121
load(":repo_utils.bzl", "repo_utils")
2222

23+
_ENABLE_PIPSTAR_ENVVAR_NAME = "RULES_PYTHON_ENABLE_PIPSTAR"
24+
_ENABLE_PIPSTAR_DEFAULT = "0"
2325
_ENABLE_PYSTAR_ENVVAR_NAME = "RULES_PYTHON_ENABLE_PYSTAR"
2426
_ENABLE_PYSTAR_DEFAULT = "1"
2527
_ENABLE_DEPRECATION_WARNINGS_ENVVAR_NAME = "RULES_PYTHON_DEPRECATION_WARNINGS"
@@ -28,6 +30,7 @@ _ENABLE_DEPRECATION_WARNINGS_DEFAULT = "0"
2830
_CONFIG_TEMPLATE = """\
2931
config = struct(
3032
enable_pystar = {enable_pystar},
33+
enable_pipstar = {enable_pipstar},
3134
enable_deprecation_warnings = {enable_deprecation_warnings},
3235
BuiltinPyInfo = getattr(getattr(native, "legacy_globals", None), "PyInfo", {builtin_py_info_symbol}),
3336
BuiltinPyRuntimeInfo = getattr(getattr(native, "legacy_globals", None), "PyRuntimeInfo", {builtin_py_runtime_info_symbol}),
@@ -84,6 +87,7 @@ def _internal_config_repo_impl(rctx):
8487

8588
rctx.file("rules_python_config.bzl", _CONFIG_TEMPLATE.format(
8689
enable_pystar = enable_pystar,
90+
enable_pipstar = _bool_from_environ(rctx, _ENABLE_PIPSTAR_ENVVAR_NAME, _ENABLE_PIPSTAR_DEFAULT),
8791
enable_deprecation_warnings = _bool_from_environ(rctx, _ENABLE_DEPRECATION_WARNINGS_ENVVAR_NAME, _ENABLE_DEPRECATION_WARNINGS_DEFAULT),
8892
builtin_py_info_symbol = builtin_py_info_symbol,
8993
builtin_py_runtime_info_symbol = builtin_py_runtime_info_symbol,

‎python/private/pypi/whl_installer/arguments.py

Copy file name to clipboardExpand all lines: python/private/pypi/whl_installer/arguments.py
+5Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ def parser(**kwargs: Any) -> argparse.ArgumentParser:
4747
type=Platform.from_string,
4848
help="Platforms to target dependencies. Can be used multiple times.",
4949
)
50+
parser.add_argument(
51+
"--enable-pipstar",
52+
action="store_true",
53+
help="Disable certain code paths if we expect to process the whl in Starlark.",
54+
)
5055
parser.add_argument(
5156
"--pip_data_exclude",
5257
action="store",

‎python/private/pypi/whl_installer/wheel_installer.py

Copy file name to clipboardExpand all lines: python/private/pypi/whl_installer/wheel_installer.py
+26-18Lines changed: 26 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ def _setup_namespace_pkg_compatibility(wheel_dir: str) -> None:
104104
def _extract_wheel(
105105
wheel_file: str,
106106
extras: Dict[str, Set[str]],
107+
enable_pipstar: bool,
107108
enable_implicit_namespace_pkgs: bool,
108109
platforms: List[wheel.Platform],
109110
installation_dir: Path = Path("."),
@@ -114,6 +115,7 @@ def _extract_wheel(
114115
wheel_file: the filepath of the .whl
115116
installation_dir: the destination directory for installation of the wheel.
116117
extras: a list of extras to add as dependencies for the installed wheel
118+
enable_pipstar: if true, turns off certain operations.
117119
enable_implicit_namespace_pkgs: if true, disables conversion of implicit namespace packages and will unzip as-is
118120
"""
119121

@@ -123,26 +125,31 @@ def _extract_wheel(
123125
if not enable_implicit_namespace_pkgs:
124126
_setup_namespace_pkg_compatibility(installation_dir)
125127

126-
extras_requested = extras[whl.name] if whl.name in extras else set()
127-
128-
dependencies = whl.dependencies(extras_requested, platforms)
128+
metadata = {
129+
"python_version": f"{sys.version_info[0]}.{sys.version_info[1]}.{sys.version_info[2]}",
130+
"entry_points": [
131+
{
132+
"name": name,
133+
"module": module,
134+
"attribute": attribute,
135+
}
136+
for name, (module, attribute) in sorted(whl.entry_points().items())
137+
],
138+
}
139+
if not enable_pipstar:
140+
extras_requested = extras[whl.name] if whl.name in extras else set()
141+
dependencies = whl.dependencies(extras_requested, platforms)
142+
143+
metadata.update(
144+
{
145+
"name": whl.name,
146+
"version": whl.version,
147+
"deps": dependencies.deps,
148+
"deps_by_platform": dependencies.deps_select,
149+
}
150+
)
129151

130152
with open(os.path.join(installation_dir, "metadata.json"), "w") as f:
131-
metadata = {
132-
"name": whl.name,
133-
"version": whl.version,
134-
"deps": dependencies.deps,
135-
"python_version": f"{sys.version_info[0]}.{sys.version_info[1]}.{sys.version_info[2]}",
136-
"deps_by_platform": dependencies.deps_select,
137-
"entry_points": [
138-
{
139-
"name": name,
140-
"module": module,
141-
"attribute": attribute,
142-
}
143-
for name, (module, attribute) in sorted(whl.entry_points().items())
144-
],
145-
}
146153
json.dump(metadata, f)
147154

148155

@@ -161,6 +168,7 @@ def main() -> None:
161168
_extract_wheel(
162169
wheel_file=whl,
163170
extras=extras,
171+
enable_pipstar=args.enable_pipstar,
164172
enable_implicit_namespace_pkgs=args.enable_implicit_namespace_pkgs,
165173
platforms=arguments.get_platforms(args),
166174
)

‎python/private/pypi/whl_library.bzl

Copy file name to clipboardExpand all lines: python/private/pypi/whl_library.bzl
+139-68Lines changed: 139 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,19 @@
1414

1515
""
1616

17+
load("@rules_python_internal//:rules_python_config.bzl", rp_config = "config")
1718
load("//python/private:auth.bzl", "AUTH_ATTRS", "get_auth")
1819
load("//python/private:envsubst.bzl", "envsubst")
1920
load("//python/private:is_standalone_interpreter.bzl", "is_standalone_interpreter")
2021
load("//python/private:repo_utils.bzl", "REPO_DEBUG_ENV_VAR", "repo_utils")
2122
load(":attrs.bzl", "ATTRS", "use_isolated")
2223
load(":deps.bzl", "all_repo_names", "record_files")
2324
load(":generate_whl_library_build_bazel.bzl", "generate_whl_library_build_bazel")
25+
load(":parse_requirements.bzl", "host_platform")
2426
load(":parse_whl_name.bzl", "parse_whl_name")
2527
load(":patch_whl.bzl", "patch_whl")
2628
load(":pypi_repo_utils.bzl", "pypi_repo_utils")
29+
load(":whl_metadata.bzl", "whl_metadata")
2730
load(":whl_target_platforms.bzl", "whl_target_platforms")
2831

2932
_CPPFLAGS = "CPPFLAGS"
@@ -340,79 +343,147 @@ def _whl_library_impl(rctx):
340343
timeout = rctx.attr.timeout,
341344
)
342345

343-
target_platforms = rctx.attr.experimental_target_platforms or []
344-
if target_platforms:
345-
parsed_whl = parse_whl_name(whl_path.basename)
346-
347-
# NOTE @aignas 2023-12-04: if the wheel is a platform specific wheel, we
348-
# only include deps for that target platform
349-
if parsed_whl.platform_tag != "any":
350-
target_platforms = [
351-
p.target_platform
352-
for p in whl_target_platforms(
353-
platform_tag = parsed_whl.platform_tag,
354-
abi_tag = parsed_whl.abi_tag.strip("tm"),
355-
)
356-
]
357-
358-
pypi_repo_utils.execute_checked(
359-
rctx,
360-
op = "whl_library.ExtractWheel({}, {})".format(rctx.attr.name, whl_path),
361-
python = python_interpreter,
362-
arguments = args + [
363-
"--whl-file",
364-
whl_path,
365-
] + ["--platform={}".format(p) for p in target_platforms],
366-
srcs = rctx.attr._python_srcs,
367-
environment = environment,
368-
quiet = rctx.attr.quiet,
369-
timeout = rctx.attr.timeout,
370-
logger = logger,
371-
)
346+
if rp_config.enable_pipstar:
347+
pypi_repo_utils.execute_checked(
348+
rctx,
349+
op = "whl_library.ExtractWheel({}, {})".format(rctx.attr.name, whl_path),
350+
python = python_interpreter,
351+
arguments = args + [
352+
"--whl-file",
353+
whl_path,
354+
"--enable-pipstar",
355+
],
356+
srcs = rctx.attr._python_srcs,
357+
environment = environment,
358+
quiet = rctx.attr.quiet,
359+
timeout = rctx.attr.timeout,
360+
logger = logger,
361+
)
372362

373-
metadata = json.decode(rctx.read("metadata.json"))
374-
rctx.delete("metadata.json")
363+
metadata = json.decode(rctx.read("metadata.json"))
364+
rctx.delete("metadata.json")
365+
python_version = metadata["python_version"]
375366

376-
# NOTE @aignas 2024-06-22: this has to live on until we stop supporting
377-
# passing `twine` as a `:pkg` library via the `WORKSPACE` builds.
378-
#
379-
# See ../../packaging.bzl line 190
380-
entry_points = {}
381-
for item in metadata["entry_points"]:
382-
name = item["name"]
383-
module = item["module"]
384-
attribute = item["attribute"]
385-
386-
# There is an extreme edge-case with entry_points that end with `.py`
387-
# See: https://github.com/bazelbuild/bazel/blob/09c621e4cf5b968f4c6cdf905ab142d5961f9ddc/src/test/java/com/google/devtools/build/lib/rules/python/PyBinaryConfiguredTargetTest.java#L174
388-
entry_point_without_py = name[:-3] + "_py" if name.endswith(".py") else name
389-
entry_point_target_name = (
390-
_WHEEL_ENTRY_POINT_PREFIX + "_" + entry_point_without_py
367+
# NOTE @aignas 2024-06-22: this has to live on until we stop supporting
368+
# passing `twine` as a `:pkg` library via the `WORKSPACE` builds.
369+
#
370+
# See ../../packaging.bzl line 190
371+
entry_points = {}
372+
for item in metadata["entry_points"]:
373+
name = item["name"]
374+
module = item["module"]
375+
attribute = item["attribute"]
376+
377+
# There is an extreme edge-case with entry_points that end with `.py`
378+
# See: https://github.com/bazelbuild/bazel/blob/09c621e4cf5b968f4c6cdf905ab142d5961f9ddc/src/test/java/com/google/devtools/build/lib/rules/python/PyBinaryConfiguredTargetTest.java#L174
379+
entry_point_without_py = name[:-3] + "_py" if name.endswith(".py") else name
380+
entry_point_target_name = (
381+
_WHEEL_ENTRY_POINT_PREFIX + "_" + entry_point_without_py
382+
)
383+
entry_point_script_name = entry_point_target_name + ".py"
384+
385+
rctx.file(
386+
entry_point_script_name,
387+
_generate_entry_point_contents(module, attribute),
388+
)
389+
entry_points[entry_point_without_py] = entry_point_script_name
390+
391+
metadata = whl_metadata(
392+
install_dir = whl_path.dirname.get_child("site-packages"),
393+
read_fn = rctx.read,
394+
logger = logger,
391395
)
392-
entry_point_script_name = entry_point_target_name + ".py"
393396

394-
rctx.file(
395-
entry_point_script_name,
396-
_generate_entry_point_contents(module, attribute),
397+
build_file_contents = generate_whl_library_build_bazel(
398+
name = whl_path.basename,
399+
dep_template = rctx.attr.dep_template or "@{}{{name}}//:{{target}}".format(rctx.attr.repo_prefix),
400+
entry_points = entry_points,
401+
metadata_name = metadata.name,
402+
metadata_version = metadata.version,
403+
default_python_version = python_version,
404+
requires_dist = metadata.requires_dist,
405+
target_platforms = rctx.attr.experimental_target_platforms or [host_platform(rctx)],
406+
# TODO @aignas 2025-04-14: load through the hub:
407+
annotation = None if not rctx.attr.annotation else struct(**json.decode(rctx.read(rctx.attr.annotation))),
408+
data_exclude = rctx.attr.pip_data_exclude,
409+
group_deps = rctx.attr.group_deps,
410+
group_name = rctx.attr.group_name,
397411
)
398-
entry_points[entry_point_without_py] = entry_point_script_name
399-
400-
build_file_contents = generate_whl_library_build_bazel(
401-
name = whl_path.basename,
402-
dep_template = rctx.attr.dep_template or "@{}{{name}}//:{{target}}".format(rctx.attr.repo_prefix),
403-
entry_points = entry_points,
404-
# TODO @aignas 2025-04-14: load through the hub:
405-
dependencies = metadata["deps"],
406-
dependencies_by_platform = metadata["deps_by_platform"],
407-
annotation = None if not rctx.attr.annotation else struct(**json.decode(rctx.read(rctx.attr.annotation))),
408-
data_exclude = rctx.attr.pip_data_exclude,
409-
group_deps = rctx.attr.group_deps,
410-
group_name = rctx.attr.group_name,
411-
tags = [
412-
"pypi_name={}".format(metadata["name"]),
413-
"pypi_version={}".format(metadata["version"]),
414-
],
415-
)
412+
else:
413+
target_platforms = rctx.attr.experimental_target_platforms or []
414+
if target_platforms:
415+
parsed_whl = parse_whl_name(whl_path.basename)
416+
417+
# NOTE @aignas 2023-12-04: if the wheel is a platform specific wheel, we
418+
# only include deps for that target platform
419+
if parsed_whl.platform_tag != "any":
420+
target_platforms = [
421+
p.target_platform
422+
for p in whl_target_platforms(
423+
platform_tag = parsed_whl.platform_tag,
424+
abi_tag = parsed_whl.abi_tag.strip("tm"),
425+
)
426+
]
427+
428+
pypi_repo_utils.execute_checked(
429+
rctx,
430+
op = "whl_library.ExtractWheel({}, {})".format(rctx.attr.name, whl_path),
431+
python = python_interpreter,
432+
arguments = args + [
433+
"--whl-file",
434+
whl_path,
435+
] + ["--platform={}".format(p) for p in target_platforms],
436+
srcs = rctx.attr._python_srcs,
437+
environment = environment,
438+
quiet = rctx.attr.quiet,
439+
timeout = rctx.attr.timeout,
440+
logger = logger,
441+
)
442+
443+
metadata = json.decode(rctx.read("metadata.json"))
444+
rctx.delete("metadata.json")
445+
446+
# NOTE @aignas 2024-06-22: this has to live on until we stop supporting
447+
# passing `twine` as a `:pkg` library via the `WORKSPACE` builds.
448+
#
449+
# See ../../packaging.bzl line 190
450+
entry_points = {}
451+
for item in metadata["entry_points"]:
452+
name = item["name"]
453+
module = item["module"]
454+
attribute = item["attribute"]
455+
456+
# There is an extreme edge-case with entry_points that end with `.py`
457+
# See: https://github.com/bazelbuild/bazel/blob/09c621e4cf5b968f4c6cdf905ab142d5961f9ddc/src/test/java/com/google/devtools/build/lib/rules/python/PyBinaryConfiguredTargetTest.java#L174
458+
entry_point_without_py = name[:-3] + "_py" if name.endswith(".py") else name
459+
entry_point_target_name = (
460+
_WHEEL_ENTRY_POINT_PREFIX + "_" + entry_point_without_py
461+
)
462+
entry_point_script_name = entry_point_target_name + ".py"
463+
464+
rctx.file(
465+
entry_point_script_name,
466+
_generate_entry_point_contents(module, attribute),
467+
)
468+
entry_points[entry_point_without_py] = entry_point_script_name
469+
470+
build_file_contents = generate_whl_library_build_bazel(
471+
name = whl_path.basename,
472+
dep_template = rctx.attr.dep_template or "@{}{{name}}//:{{target}}".format(rctx.attr.repo_prefix),
473+
entry_points = entry_points,
474+
# TODO @aignas 2025-04-14: load through the hub:
475+
dependencies = metadata["deps"],
476+
dependencies_by_platform = metadata["deps_by_platform"],
477+
annotation = None if not rctx.attr.annotation else struct(**json.decode(rctx.read(rctx.attr.annotation))),
478+
data_exclude = rctx.attr.pip_data_exclude,
479+
group_deps = rctx.attr.group_deps,
480+
group_name = rctx.attr.group_name,
481+
tags = [
482+
"pypi_name={}".format(metadata["name"]),
483+
"pypi_version={}".format(metadata["version"]),
484+
],
485+
)
486+
416487
rctx.file("BUILD.bazel", build_file_contents)
417488

418489
return

‎tests/pypi/whl_installer/wheel_installer_test.py

Copy file name to clipboardExpand all lines: tests/pypi/whl_installer/wheel_installer_test.py
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ def test_wheel_exists(self) -> None:
7272
extras={},
7373
enable_implicit_namespace_pkgs=False,
7474
platforms=[],
75+
enable_pipstar = False,
7576
)
7677

7778
want_files = [

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.