-pip_import(name, extra_pip_args, python_interpreter, python_interpreter_target, requirements, timeout) -- -A rule for importing `requirements.txt` dependencies into Bazel. - -This rule imports a `requirements.txt` file and generates a new -`requirements.bzl` file. This is used via the `WORKSPACE` pattern: - -```python -pip_import( - name = "foo", - requirements = ":requirements.txt", -) -load("@foo//:requirements.bzl", "pip_install") -pip_install() -``` - -You can then reference imported dependencies from your `BUILD` file with: - -```python -load("@foo//:requirements.bzl", "requirement") -py_library( - name = "bar", - ... - deps = [ - "//my/other:dep", - requirement("futures"), - requirement("mock"), - ], -) -``` - -Or alternatively: -```python -load("@foo//:requirements.bzl", "all_requirements") -py_binary( - name = "baz", - ... - deps = [ - ":foo", - ] + all_requirements, -) -``` - - -### Attributes - -
name |
-
- Name; required
- - A unique name for this repository. - - |
-
extra_pip_args |
-
- List of strings; optional
- - Extra arguments to pass on to pip. Must not contain spaces. - - |
-
python_interpreter |
-
- String; optional
- - The command to run the Python interpreter used to invoke pip and unpack the -wheels. - - |
-
python_interpreter_target |
-
- Label; optional
- - If you are using a custom python interpreter built by another repository rule, -use this attribute to specify its BUILD target. This allows pip_import to invoke -pip using the same interpreter as your toolchain. If set, takes precedence over -python_interpreter. - - |
-
requirements |
-
- Label; required
- - The label of the requirements.txt file. - - |
-
timeout |
-
- Integer; optional
- - Timeout (in seconds) for repository fetch. - - |
-
-pip3_import(kwargs) -- -A wrapper around pip_import that uses the `python3` system command. - -Use this for requirements of PY3 programs. - -### Parameters - -
kwargs |
- - optional. - | -
-pip_repositories() -- -Pull in dependencies needed to use the packaging rules. - - - diff --git a/docs/precompiling.md b/docs/precompiling.md new file mode 100644 index 0000000000..a46608f77e --- /dev/null +++ b/docs/precompiling.md @@ -0,0 +1,124 @@ +# Precompiling + +Precompiling is compiling Python source files (`.py` files) into byte code +(`.pyc` files) at build time instead of runtime. Doing it at build time can +improve performance by skipping that work at runtime. + +Precompiling is disabled by default, so you must enable it using flags or +attributes to use it. + +## Overhead of precompiling + +While precompiling helps runtime performance, it has two main costs: +1. Increasing the size (count and disk usage) of runfiles. It approximately + double the count of the runfiles because for every `.py` file, there is also + a `.pyc` file. Compiled files are generally around the same size as the + source files, so it approximately doubles the disk usage. +2. Precompiling requires running an extra action at build time. While + compiling itself isn't that expensive, the overhead can become noticable + as more files need to be compiled. + +## Binary-level opt-in + +Binary-level opt-in allows enabling precompiling on a per-target basic. This is +useful for situations such as: + +* Globally enabling precompiling in your `.bazelrc` isn't feasible. This may + be because some targets don't work with precompiling, e.g. because they're too + big. +* Enabling precompiling for build tools (exec config targets) separately from + target-config programs. + +To use this approach, set the {bzl:attr}`pyc_collection` attribute on the +binaries/tests that should or should not use precompiling. Then change the +{bzl:flag}`--precompile` default. + +The default for the {bzl:attr}`pyc_collection` attribute is controlled by the flag +{bzl:obj}`--@rules_python//python/config_settings:precompile`, so you +can use an opt-in or opt-out approach by setting its value: +* targets must opt-out: `--@rules_python//python/config_settings:precompile=enabled` +* targets must opt-in: `--@rules_python//python/config_settings:precompile=disabled` + +## Pyc-only builds + +A pyc-only build (aka "source less" builds) is when only `.pyc` files are +included; the source `.py` files are not included. + +To enable this, set +{bzl:obj}`--@rules_python//python/config_settings:precompile_source_retention=omit_source` +flag on the command line or the {bzl:attr}`precompile_source_retention=omit_source` +attribute on specific targets. + +The advantage of pyc-only builds are: +* Fewer total files in a binary. +* Imports _may_ be _slightly_ faster. + +The disadvantages are: +* Error messages will be less precise because the precise line and offset + information isn't in an pyc file. +* pyc files are Python major-version specific. + +:::{note} +pyc files are not a form of hiding source code. They are trivial to uncompile, +and uncompiling them can recover almost the original source. +::: + +## Advanced precompiler customization + +The default implementation of the precompiler is a persistent, multiplexed, +sandbox-aware, cancellation-enabled, json-protocol worker that uses the same +interpreter as the target toolchain. This works well for local builds, but may +not work as well for remote execution builds. To customize the precompiler, two +mechanisms are available: + +* The exec tools toolchain allows customizing the precompiler binary used with + the {bzl:attr}`precompiler` attribute. Arbitrary binaries are supported. +* The execution requirements can be customized using + `--@rules_python//tools/precompiler:execution_requirements`. This is a list + flag that can be repeated. Each entry is a key=value that is added to the + execution requirements of the `PyCompile` action. Note that this flag + is specific to the rules_python precompiler. If a custom binary is used, + this flag will have to be propagated from the custom binary using the + `testing.ExecutionInfo` provider; refer to the `py_interpreter_program` an + +The default precompiler implementation is an asynchronous/concurrent +implementation. If you find it has bugs or hangs, please report them. In the +meantime, the flag `--worker_extra_flag=PyCompile=--worker_impl=serial` can +be used to switch to a synchronous/serial implementation that may not perform +as well, but is less likely to have issues. + +The `execution_requirements` keys of most relevance are: +* `supports-workers`: 1 or 0, to indicate if a regular persistent worker is + desired. +* `supports-multiplex-workers`: 1 o 0, to indicate if a multiplexed persistent + worker is desired. +* `requires-worker-protocol`: json or proto; the rules_python precompiler + currently only supports json. +* `supports-multiplex-sandboxing`: 1 or 0, to indicate if sanboxing is of the + worker is supported. +* `supports-worker-cancellation`: 1 or 1, to indicate if requests to the worker + can be cancelled. + +Note that any execution requirements values can be specified in the flag. + +## Known issues, caveats, and idiosyncracies + +* Precompiling requires Bazel 7+ with the Pystar rule implementation enabled. +* Mixing rules_python PyInfo with Bazel builtin PyInfo will result in pyc files + being dropped. +* Precompiled files may not be used in certain cases prior to Python 3.11. This + occurs due to Python adding the directory of the binary's main `.py` file, which + causes the module to be found in the workspace source directory instead of + within the binary's runfiles directory (where the pyc files are). This can + usually be worked around by removing `sys.path[0]` (or otherwise ensuring the + runfiles directory comes before the repos source directory in `sys.path`). +* The pyc filename does not include the optimization level (e.g. + `foo.cpython-39.opt-2.pyc`). This works fine (it's all byte code), but also + means the interpreter `-O` argument can't be used -- doing so will cause the + interpreter to look for the non-existent `opt-N` named files. +* Targets with the same source files and different exec properites will result + in action conflicts. This most commonly occurs when a `py_binary` and + `py_library` have the same source files. To fix, modify both targets so + they have the same exec properties. If this is difficult because unsupported + exec groups end up being passed to the Python rules, please file an issue + to have those exec groups added to the Python rules. diff --git a/docs/pypi/circular-dependencies.md b/docs/pypi/circular-dependencies.md new file mode 100644 index 0000000000..d22f5b36a7 --- /dev/null +++ b/docs/pypi/circular-dependencies.md @@ -0,0 +1,82 @@ +:::{default-domain} bzl +::: + +# Circular dependencies + +Sometimes PyPi packages contain dependency cycles -- for instance a particular +version `sphinx` (this is no longer the case in the latest version as of +2024-06-02) depends on `sphinxcontrib-serializinghtml`. When using them as +`requirement()`s, ala + +```starlark +py_binary( + name = "doctool", + ... + deps = [ + requirement("sphinx"), + ], +) +``` + +Bazel will protest because it doesn't support cycles in the build graph -- + +``` +ERROR: .../external/pypi_sphinxcontrib_serializinghtml/BUILD.bazel:44:6: in alias rule @pypi_sphinxcontrib_serializinghtml//:pkg: cycle in dependency graph: + //:doctool (...) + @pypi//sphinxcontrib_serializinghtml:pkg (...) +.-> @pypi_sphinxcontrib_serializinghtml//:pkg (...) +| @pypi_sphinxcontrib_serializinghtml//:_pkg (...) +| @pypi_sphinx//:pkg (...) +| @pypi_sphinx//:_pkg (...) +`-- @pypi_sphinxcontrib_serializinghtml//:pkg (...) +``` + +The `experimental_requirement_cycles` attribute allows you to work around these +issues by specifying groups of packages which form cycles. `pip_parse` will +transparently fix the cycles for you and provide the cyclic dependencies +simultaneously. + +```starlark + ... + experimental_requirement_cycles = { + "sphinx": [ + "sphinx", + "sphinxcontrib-serializinghtml", + ] + }, +) +``` + +`pip_parse` supports fixing multiple cycles simultaneously, however cycles must +be distinct. `apache-airflow` for instance has dependency cycles with a number +of its optional dependencies, which means those optional dependencies must all +be a part of the `airflow` cycle. For instance -- + +```starlark + ... + experimental_requirement_cycles = { + "airflow": [ + "apache-airflow", + "apache-airflow-providers-common-sql", + "apache-airflow-providers-postgres", + "apache-airflow-providers-sqlite", + ] + } +) +``` + +Alternatively, one could resolve the cycle by removing one leg of it. + +For example while `apache-airflow-providers-sqlite` is "baked into" the Airflow +package, `apache-airflow-providers-postgres` is not and is an optional feature. +Rather than listing `apache-airflow[postgres]` in your `requirements.txt` which +would expose a cycle via the extra, one could either _manually_ depend on +`apache-airflow` and `apache-airflow-providers-postgres` separately as +requirements. Bazel rules which need only `apache-airflow` can take it as a +dependency, and rules which explicitly want to mix in +`apache-airflow-providers-postgres` now can. + +Alternatively, one could use `rules_python`'s patching features to remove one +leg of the dependency manually. For instance by making +`apache-airflow-providers-postgres` not explicitly depend on `apache-airflow` or +perhaps `apache-airflow-providers-common-sql`. diff --git a/docs/pypi/download-workspace.md b/docs/pypi/download-workspace.md new file mode 100644 index 0000000000..48710095a4 --- /dev/null +++ b/docs/pypi/download-workspace.md @@ -0,0 +1,107 @@ +:::{default-domain} bzl +::: + +# Download (WORKSPACE) + +This documentation page covers how to download the PyPI dependencies in the legacy `WORKSPACE` setup. + +To add pip dependencies to your `WORKSPACE`, load the `pip_parse` function and +call it to create the central external repo and individual wheel external repos. + +```starlark +load("@rules_python//python:pip.bzl", "pip_parse") + +# Create a central repo that knows about the dependencies needed from +# requirements_lock.txt. +pip_parse( + name = "my_deps", + requirements_lock = "//path/to:requirements_lock.txt", +) + +# Load the starlark macro, which will define your dependencies. +load("@my_deps//:requirements.bzl", "install_deps") + +# Call it to define repos for your requirements. +install_deps() +``` + +## Interpreter selection + +Note that pip parse runs before the Bazel before decides which Python toolchain to use, it cannot +enforce that the interpreter used to invoke `pip` matches the interpreter used to run `py_binary` +targets. By default, `pip_parse` uses the system command `"python3"`. To override this, pass in the +{attr}`pip_parse.python_interpreter` attribute or {attr}`pip_parse.python_interpreter_target`. + +You can have multiple `pip_parse`s in the same workspace. This configuration will create multiple +external repos that have no relation to one another and may result in downloading the same wheels +numerous times. + +As with any repository rule, if you would like to ensure that `pip_parse` is +re-executed to pick up a non-hermetic change to your environment (e.g., updating +your system `python` interpreter), you can force it to re-execute by running +`bazel sync --only [pip_parse name]`. + +(per-os-arch-requirements)= +## Requirements for a specific OS/Architecture + +In some cases you may need to use different requirements files for different OS, Arch combinations. +This is enabled via the {attr}`pip_parse.requirements_by_platform` attribute. The keys of the +dictionary are labels to the file and the values are a list of comma separated target (os, arch) +tuples. + +For example: +```starlark + # ... + requirements_by_platform = { + "requirements_linux_x86_64.txt": "linux_x86_64", + "requirements_osx.txt": "osx_*", + "requirements_linux_exotic.txt": "linux_exotic", + "requirements_some_platforms.txt": "linux_aarch64,windows_*", + }, + # For the list of standard platforms that the rules_python has toolchains for, default to + # the following requirements file. + requirements_lock = "requirements_lock.txt", +``` + +In case of duplicate platforms, `rules_python` will raise an error as there has +to be unambiguous mapping of the requirement files to the (os, arch) tuples. + +An alternative way is to use per-OS requirement attributes. +```starlark + # ... + requirements_windows = "requirements_windows.txt", + requirements_darwin = "requirements_darwin.txt", + # For the remaining platforms (which is basically only linux OS), use this file. + requirements_lock = "requirements_lock.txt", +) +``` + +:::{note} +If you are using a universal lock file but want to restrict the list of platforms that +the lock file will be evaluated against, consider using the aforementioned +`requirements_by_platform` attribute and listing the platforms explicitly. +::: + +(vendoring-requirements)= +## Vendoring the requirements.bzl file + +:::{note} +For `bzlmod`, refer to standard `bazel vendor` usage if you want to really vendor it, otherwise +just use the `pip` extension as you would normally. + +However, be aware that there are caveats when doing so. +::: + +In some cases you may not want to generate the requirements.bzl file as a repository rule +while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module +such as a ruleset, you may want to include the `requirements.bzl` file rather than make your users +install the `WORKSPACE` setup to generate it, see {gh-issue}`608`. + +This is the same workflow as Gazelle, which creates `go_repository` rules with +[`update-repos`](https://github.com/bazelbuild/bazel-gazelle#update-repos) + +To do this, use the "write to source file" pattern documented in +
-py_runtime_pair(name, py2_runtime, py3_runtime) -- -A toolchain rule for Python. - -This wraps up to two Python runtimes, one for Python 2 and one for Python 3. -The rule consuming this toolchain will choose which runtime is appropriate. -Either runtime may be omitted, in which case the resulting toolchain will be -unusable for building Python code using that version. - -Usually the wrapped runtimes are declared using the `py_runtime` rule, but any -rule returning a `PyRuntimeInfo` provider may be used. - -This rule returns a `platform_common.ToolchainInfo` provider with the following -schema: - -```python -platform_common.ToolchainInfo( - py2_runtime =
name |
-
- Name; required
- - A unique name for this target. - - |
-
py2_runtime |
-
- Label; optional
- - The runtime to use for Python 2 targets. Must have `python_version` set to -`PY2`. - - |
-
py3_runtime |
-
- Label; optional
- - The runtime to use for Python 3 targets. Must have `python_version` set to -`PY3`. - - |
-
-py_binary(attrs) -- -See the Bazel core [py_binary](https://docs.bazel.build/versions/master/be/python.html#py_binary) documentation. - -### Parameters - -
attrs |
-
- optional.
- - Rule attributes - - |
-
-py_library(attrs) -- -See the Bazel core [py_library](https://docs.bazel.build/versions/master/be/python.html#py_library) documentation. - -### Parameters - -
attrs |
-
- optional.
- - Rule attributes - - |
-
-py_runtime(attrs) -- -See the Bazel core [py_runtime](https://docs.bazel.build/versions/master/be/python.html#py_runtime) documentation. - -### Parameters - -
attrs |
-
- optional.
- - Rule attributes - - |
-
-py_test(attrs) -- -See the Bazel core [py_test](https://docs.bazel.build/versions/master/be/python.html#py_test) documentation. - -### Parameters - -
attrs |
-
- optional.
- - Rule attributes - - |
-
-whl_library(name, extras, python_interpreter, requirements, whl) -- -A rule for importing `.whl` dependencies into Bazel. - -This rule is currently used to implement `pip_import`. It is not intended to -work standalone, and the interface may change. See `pip_import` for proper -usage. - -This rule imports a `.whl` file as a `py_library`: -```python -whl_library( - name = "foo", - whl = ":my-whl-file", - requirements = "name of pip_import rule", -) -``` - -This rule defines `@foo//:pkg` as a `py_library` target. - - -### Attributes - -
name |
-
- Name; required
- - A unique name for this repository. - - |
-
extras |
-
- List of strings; optional
-
- A subset of the "extras" available from this |
-
python_interpreter |
-
- String; optional
- - The command to run the Python interpreter used when unpacking the wheel. - - |
-
requirements |
-
- String; optional
-
- The name of the |
-
whl |
-
- Label; required
-
- The path to the |
-
-# Package just a specific py_libraries, without their dependencies
-py_wheel(
- name = "minimal_with_py_library",
- # Package data. We're building "example_minimal_library-0.0.1-py3-none-any.whl"
- distribution = "example_minimal_library",
- python_tag = "py3",
- version = "0.0.1",
- deps = [
- "//examples/wheel/lib:module_with_data",
- "//examples/wheel/lib:simple_module",
- ],
-)
-# Use py_package to collect all transitive dependencies of a target,
-# selecting just the files within a specific python package.
-py_package(
- name = "example_pkg",
- # Only include these Python packages.
- packages = ["examples.wheel"],
- deps = [":main"],
-)
+ _py_wheel(
+ name = name,
+ tags = tags,
+ **kwargs
+ )
-py_wheel(
- name = "minimal_with_py_package",
- # Package data. We're building "example_minimal_package-0.0.1-py3-none-any.whl"
- distribution = "example_minimal_package",
- python_tag = "py3",
- version = "0.0.1",
- deps = [":example_pkg"],
-)
-
-""",
- attrs = _concat_dicts(
- {
- "deps": attr.label_list(
- doc = """\
-Targets to be included in the distribution.
-
-The targets to package are usually `py_library` rules or filesets (for packaging data files).
-
-Note it's usually better to package `py_library` targets and use
-`entry_points` attribute to specify `console_scripts` than to package
-`py_binary` rules. `py_binary` targets would wrap a executable script that
-tries to locate `.runfiles` directory which is not packaged in the wheel.
-""",
- ),
- "_wheelmaker": attr.label(
- executable = True,
- cfg = "host",
- default = "//tools:wheelmaker",
- ),
- },
- _distribution_attrs,
- _requirement_attrs,
- _entrypoint_attrs,
- _other_attrs,
- ),
-)
+ twine_args = []
+ if twine or twine_binary:
+ twine_args = ["upload"]
+ twine_args.extend(publish_args)
+ twine_args.append("$(rootpath :{})/*".format(dist_target))
+
+ if twine_binary:
+ native_binary(
+ name = "{}.publish".format(name),
+ src = twine_binary,
+ out = select({
+ "@platforms//os:windows": "{}.publish_script.exe".format(name),
+ "//conditions:default": "{}.publish_script".format(name),
+ }),
+ args = twine_args,
+ data = [dist_target],
+ tags = manual_tags,
+ visibility = kwargs.get("visibility"),
+ **copy_propagating_kwargs(kwargs)
+ )
+ elif twine:
+ if not twine.endswith(":pkg"):
+ fail("twine label should look like @my_twine_repo//:pkg")
+
+ twine_main = twine.replace(":pkg", ":rules_python_wheel_entry_point_twine.py")
+
+ py_binary(
+ name = "{}.publish".format(name),
+ srcs = [twine_main],
+ args = twine_args,
+ data = [dist_target],
+ imports = ["."],
+ main = twine_main,
+ deps = [twine],
+ tags = manual_tags,
+ visibility = kwargs.get("visibility"),
+ **copy_propagating_kwargs(kwargs)
+ )
+
+py_wheel_rule = _py_wheel
diff --git a/python/pip.bzl b/python/pip.bzl
index 5027666806..44ee69d65b 100644
--- a/python/pip.bzl
+++ b/python/pip.bzl
@@ -11,69 +11,39 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Import pip requirements into Bazel."""
-
-load("//python/pip_install:pip_repository.bzl", "pip_repository")
-load("//python/pip_install:repositories.bzl", "pip_install_dependencies")
-
-def pip_install(requirements, name = "pip", **kwargs):
- """Imports a `requirements.txt` file and generates a new `requirements.bzl` file.
-
- This is used via the `WORKSPACE` pattern:
-
- ```python
- pip_install(
- requirements = ":requirements.txt",
- )
- ```
-
- You can then reference imported dependencies from your `BUILD` file with:
-
- ```python
- load("@pip//:requirements.bzl", "requirement")
- py_library(
- name = "bar",
- ...
- deps = [
- "//my/other:dep",
- requirement("requests"),
- requirement("numpy"),
- ],
- )
- ```
-
- Args:
- requirements: A 'requirements.txt' pip requirements file.
- name: A unique name for the created external repository (default 'pip').
- **kwargs: Keyword arguments passed directly to the `pip_repository` repository rule.
- """
-
- # Just in case our dependencies weren't already fetched
- pip_install_dependencies()
-
- pip_repository(
- name = name,
- requirements = requirements,
- **kwargs
- )
-
-def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs):
- # Just in case our dependencies weren't already fetched
- pip_install_dependencies()
-
- pip_repository(
- name = name,
- requirements_lock = requirements_lock,
- incremental = True,
- **kwargs
- )
-
-def pip_repositories():
- # buildifier: disable=print
- print("DEPRECATED: the pip_repositories rule has been replaced with pip_install, please see rules_python 0.1 release notes")
-
-def pip_import(**kwargs):
- fail("=" * 79 + """\n
- pip_import has been replaced with pip_install, please see the rules_python 0.1 release notes.
- To continue using it, you can load from "@rules_python//python/legacy_pip_import:pip.bzl"
- """)
+"""Rules for pip integration.
+
+This contains a set of rules that are used to support inclusion of third-party
+dependencies via fully locked `requirements.txt` files. Some of the exported
+symbols should not be used and they are either undocumented here or marked as
+for internal use only.
+
+If you are using a bazel version 7 or above with `bzlmod`, you should only care
+about the {bzl:obj}`compile_pip_requirements` macro exposed in this file. The
+rest of the symbols are for legacy `WORKSPACE` setups.
+"""
+
+load("//python/private:normalize_name.bzl", "normalize_name")
+load("//python/private/pypi:multi_pip_parse.bzl", _multi_pip_parse = "multi_pip_parse")
+load("//python/private/pypi:package_annotation.bzl", _package_annotation = "package_annotation")
+load("//python/private/pypi:pip_compile.bzl", "pip_compile")
+load("//python/private/pypi:pip_repository.bzl", "pip_repository")
+load("//python/private/pypi:whl_library_alias.bzl", _whl_library_alias = "whl_library_alias")
+load("//python/private/whl_filegroup:whl_filegroup.bzl", _whl_filegroup = "whl_filegroup")
+
+compile_pip_requirements = pip_compile
+package_annotation = _package_annotation
+pip_parse = pip_repository
+whl_filegroup = _whl_filegroup
+
+# Extra utilities visible to rules_python users.
+pip_utils = struct(
+ normalize_name = normalize_name,
+)
+
+# The following are only exported here because they are used from
+# multi_toolchain_aliases repository_rule, not intended for public use.
+#
+# See ./private/toolchains_repo.bzl
+multi_pip_parse = _multi_pip_parse
+whl_library_alias = _whl_library_alias
diff --git a/python/pip_install/BUILD b/python/pip_install/BUILD
deleted file mode 100644
index afcbcd4f17..0000000000
--- a/python/pip_install/BUILD
+++ /dev/null
@@ -1,29 +0,0 @@
-exports_files(["pip_compile.py"])
-
-filegroup(
- name = "distribution",
- srcs = glob(["*.bzl"]) + [
- "BUILD",
- "pip_compile.py",
- "//python/pip_install/extract_wheels:distribution",
- "//python/pip_install/parse_requirements_to_bzl:distribution",
- ],
- visibility = ["//:__pkg__"],
-)
-
-filegroup(
- name = "bzl",
- srcs = [
- "pip_repository.bzl",
- "repositories.bzl",
- ],
- visibility = ["//:__pkg__"],
-)
-
-exports_files(
- [
- "pip_repository.bzl",
- "repositories.bzl",
- ],
- visibility = ["//docs:__pkg__"],
-)
diff --git a/python/pip_install/BUILD.bazel b/python/pip_install/BUILD.bazel
new file mode 100644
index 0000000000..09bc46eea7
--- /dev/null
+++ b/python/pip_install/BUILD.bazel
@@ -0,0 +1,53 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+
+package(
+ default_visibility = ["//:__subpackages__"],
+)
+
+bzl_library(
+ name = "pip_repository_bzl",
+ srcs = ["pip_repository.bzl"],
+ deps = [
+ "//python/private/pypi:group_library_bzl",
+ "//python/private/pypi:package_annotation_bzl",
+ "//python/private/pypi:pip_repository_bzl",
+ "//python/private/pypi:whl_library_bzl",
+ ],
+)
+
+bzl_library(
+ name = "requirements_bzl",
+ srcs = ["requirements.bzl"],
+ deps = ["//python/private/pypi:pip_compile_bzl"],
+)
+
+filegroup(
+ name = "distribution",
+ srcs = glob(["**"]),
+ visibility = ["//python:__pkg__"],
+)
+
+filegroup(
+ name = "bzl",
+ srcs = glob(["*.bzl"]),
+ visibility = ["//:__subpackages__"],
+)
+
+exports_files(
+ glob(["*.bzl"]),
+ visibility = ["//docs:__pkg__"],
+)
diff --git a/python/pip_install/extract_wheels/BUILD b/python/pip_install/extract_wheels/BUILD
deleted file mode 100644
index 92a0c7a9d9..0000000000
--- a/python/pip_install/extract_wheels/BUILD
+++ /dev/null
@@ -1,19 +0,0 @@
-load("@rules_python//python:defs.bzl", "py_binary")
-
-py_binary(
- name = "extract_wheels",
- srcs = [
- "__init__.py",
- "__main__.py",
- ],
- main = "__main__.py",
- deps = ["//python/pip_install/extract_wheels/lib"],
-)
-
-filegroup(
- name = "distribution",
- srcs = glob(["*"]) + [
- "//python/pip_install/extract_wheels/lib:distribution",
- ],
- visibility = ["//python/pip_install:__subpackages__"],
-)
diff --git a/python/pip_install/extract_wheels/__init__.py b/python/pip_install/extract_wheels/__init__.py
deleted file mode 100644
index 96913cdad7..0000000000
--- a/python/pip_install/extract_wheels/__init__.py
+++ /dev/null
@@ -1,94 +0,0 @@
-"""extract_wheels
-
-extract_wheels resolves and fetches artifacts transitively from the Python Package Index (PyPI) based on a
-requirements.txt. It generates the required BUILD files to consume these packages as Python libraries.
-
-Under the hood, it depends on the `pip wheel` command to do resolution, download, and compilation into wheels.
-"""
-import argparse
-import glob
-import os
-import subprocess
-import sys
-import json
-
-from python.pip_install.extract_wheels.lib import bazel, requirements, arguments
-
-
-def configure_reproducible_wheels() -> None:
- """Modifies the environment to make wheel building reproducible.
-
- Wheels created from sdists are not reproducible by default. We can however workaround this by
- patching in some configuration with environment variables.
- """
-
- # wheel, by default, enables debug symbols in GCC. This incidentally captures the build path in the .so file
- # We can override this behavior by disabling debug symbols entirely.
- # https://github.com/pypa/pip/issues/6505
- if "CFLAGS" in os.environ:
- os.environ["CFLAGS"] += " -g0"
- else:
- os.environ["CFLAGS"] = "-g0"
-
- # set SOURCE_DATE_EPOCH to 1980 so that we can use python wheels
- # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl
- if "SOURCE_DATE_EPOCH" not in os.environ:
- os.environ["SOURCE_DATE_EPOCH"] = "315532800"
-
- # Python wheel metadata files can be unstable.
- # See https://bitbucket.org/pypa/wheel/pull-requests/74/make-the-output-of-metadata-files/diff
- if "PYTHONHASHSEED" not in os.environ:
- os.environ["PYTHONHASHSEED"] = "0"
-
-
-def main() -> None:
- """Main program.
-
- Exits zero on successful program termination, non-zero otherwise.
- """
-
- configure_reproducible_wheels()
-
- parser = argparse.ArgumentParser(
- description="Resolve and fetch artifacts transitively from PyPI"
- )
- parser.add_argument(
- "--requirements",
- action="store",
- required=True,
- help="Path to requirements.txt from where to install dependencies",
- )
- arguments.parse_common_args(parser)
- args = parser.parse_args()
-
- pip_args = [sys.executable, "-m", "pip", "--isolated", "wheel", "-r", args.requirements]
- if args.extra_pip_args:
- pip_args += json.loads(args.extra_pip_args)["args"]
-
- # Assumes any errors are logged by pip so do nothing. This command will fail if pip fails
- subprocess.run(pip_args, check=True)
-
- extras = requirements.parse_extras(args.requirements)
-
- if args.pip_data_exclude:
- pip_data_exclude = json.loads(args.pip_data_exclude)["exclude"]
- else:
- pip_data_exclude = []
-
- repo_label = "@%s" % args.repo
-
- targets = [
- '"%s%s"'
- % (
- repo_label,
- bazel.extract_wheel(
- whl, extras, pip_data_exclude, args.enable_implicit_namespace_pkgs
- ),
- )
- for whl in glob.glob("*.whl")
- ]
-
- with open("requirements.bzl", "w") as requirement_file:
- requirement_file.write(
- bazel.generate_requirements_file_contents(repo_label, targets)
- )
diff --git a/python/pip_install/extract_wheels/__main__.py b/python/pip_install/extract_wheels/__main__.py
deleted file mode 100644
index 40b690e2fb..0000000000
--- a/python/pip_install/extract_wheels/__main__.py
+++ /dev/null
@@ -1,5 +0,0 @@
-"""Main entry point."""
-from python.pip_install.extract_wheels import main
-
-if __name__ == "__main__":
- main()
diff --git a/python/pip_install/extract_wheels/lib/BUILD b/python/pip_install/extract_wheels/lib/BUILD
deleted file mode 100644
index 82c7173982..0000000000
--- a/python/pip_install/extract_wheels/lib/BUILD
+++ /dev/null
@@ -1,92 +0,0 @@
-load("@rules_python//python:defs.bzl", "py_library", "py_test")
-load("//python/pip_install:repositories.bzl", "requirement")
-
-py_library(
- name = "lib",
- srcs = [
- "arguments.py",
- "bazel.py",
- "namespace_pkgs.py",
- "purelib.py",
- "requirements.py",
- "wheel.py",
- ],
- visibility = [
- "//python/pip_install/extract_wheels:__subpackages__",
- "//python/pip_install/parse_requirements_to_bzl:__subpackages__",
- ],
- deps = [
- requirement("pkginfo"),
- requirement("setuptools"),
- ],
-)
-
-py_test(
- name = "namespace_pkgs_test",
- size = "small",
- srcs = [
- "namespace_pkgs_test.py",
- ],
- tags = ["unit"],
- deps = [
- ":lib",
- ],
-)
-
-py_test(
- name = "requirements_test",
- size = "small",
- srcs = [
- "requirements_test.py",
- ],
- tags = ["unit"],
- deps = [
- ":lib",
- ],
-)
-
-py_test(
- name = "arguments_test",
- size = "small",
- srcs = [
- "arguments_test.py",
- ],
- tags = ["unit"],
- deps = [
- ":lib",
- "//python/pip_install/parse_requirements_to_bzl:lib",
- ],
-)
-
-py_test(
- name = "whl_filegroup_test",
- size = "small",
- srcs = [
- "whl_filegroup_test.py",
- ],
- data = ["//examples/wheel:minimal_with_py_package"],
- tags = ["unit"],
- deps = [
- ":lib",
- ],
-)
-
-py_test(
- name = "requirements_bzl_test",
- size = "small",
- srcs = [
- "requirements_bzl_test.py",
- ],
- deps = [
- ":lib",
- ],
-)
-
-filegroup(
- name = "distribution",
- srcs = glob(
- ["*"],
- exclude = ["*_test.py"],
- ),
- visibility = ["//python/pip_install:__subpackages__"],
-)
diff --git a/python/pip_install/extract_wheels/lib/arguments.py b/python/pip_install/extract_wheels/lib/arguments.py
deleted file mode 100644
index ee9a6491bc..0000000000
--- a/python/pip_install/extract_wheels/lib/arguments.py
+++ /dev/null
@@ -1,24 +0,0 @@
-from argparse import ArgumentParser
-
-
-def parse_common_args(parser: ArgumentParser) -> ArgumentParser:
- parser.add_argument(
- "--repo",
- action="store",
- required=True,
- help="The external repo name to install dependencies. In the format '@{REPO_NAME}'",
- )
- parser.add_argument(
- "--extra_pip_args", action="store", help="Extra arguments to pass down to pip.",
- )
- parser.add_argument(
- "--pip_data_exclude",
- action="store",
- help="Additional data exclusion parameters to add to the pip packages BUILD file.",
- )
- parser.add_argument(
- "--enable_implicit_namespace_pkgs",
- action="store_true",
- help="Disables conversion of implicit namespace packages into pkg-util style packages.",
- )
- return parser
diff --git a/python/pip_install/extract_wheels/lib/arguments_test.py b/python/pip_install/extract_wheels/lib/arguments_test.py
deleted file mode 100644
index c0338bd0ca..0000000000
--- a/python/pip_install/extract_wheels/lib/arguments_test.py
+++ /dev/null
@@ -1,27 +0,0 @@
-import argparse
-import json
-import unittest
-
-from python.pip_install.extract_wheels.lib import arguments
-from python.pip_install.parse_requirements_to_bzl import deserialize_structured_args
-
-
-class ArgumentsTestCase(unittest.TestCase):
- def test_arguments(self) -> None:
- parser = argparse.ArgumentParser()
- parser = arguments.parse_common_args(parser)
- repo_name = "foo"
- index_url = "--index_url=pypi.org/simple"
- args_dict = vars(parser.parse_args(
- args=["--repo", repo_name, "--extra_pip_args={index_url}".format(index_url=json.dumps({"args": index_url}))]))
- args_dict = deserialize_structured_args(args_dict)
- self.assertIn("repo", args_dict)
- self.assertIn("extra_pip_args", args_dict)
- self.assertEqual(args_dict["pip_data_exclude"], [])
- self.assertEqual(args_dict["enable_implicit_namespace_pkgs"], False)
- self.assertEqual(args_dict["repo"], repo_name)
- self.assertEqual(args_dict["extra_pip_args"], index_url)
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/extract_wheels/lib/bazel.py b/python/pip_install/extract_wheels/lib/bazel.py
deleted file mode 100644
index 94c681c50a..0000000000
--- a/python/pip_install/extract_wheels/lib/bazel.py
+++ /dev/null
@@ -1,254 +0,0 @@
-"""Utility functions to manipulate Bazel files"""
-import os
-import textwrap
-import json
-from typing import Iterable, List, Dict, Set, Optional
-import shutil
-
-from python.pip_install.extract_wheels.lib import namespace_pkgs, wheel, purelib
-
-
-WHEEL_FILE_LABEL = "whl"
-PY_LIBRARY_LABEL = "pkg"
-
-
-def generate_build_file_contents(
- name: str, dependencies: List[str], whl_file_deps: List[str], pip_data_exclude: List[str],
-) -> str:
- """Generate a BUILD file for an unzipped Wheel
-
- Args:
- name: the target name of the py_library
- dependencies: a list of Bazel labels pointing to dependencies of the library
- whl_file_deps: a list of Bazel labels pointing to wheel file dependencies of this wheel.
-
- Returns:
- A complete BUILD file as a string
-
- We allow for empty Python sources as for Wheels containing only compiled C code
- there may be no Python sources whatsoever (e.g. packages written in Cython: like `pymssql`).
- """
-
- data_exclude = ["*.whl", "**/*.py", "**/* *", "BUILD.bazel", "WORKSPACE"] + pip_data_exclude
-
- return textwrap.dedent(
- """\
- package(default_visibility = ["//visibility:public"])
-
- load("@rules_python//python:defs.bzl", "py_library")
-
- filegroup(
- name="{whl_file_label}",
- srcs=glob(["*.whl"]),
- data=[{whl_file_deps}]
- )
-
- py_library(
- name = "{name}",
- srcs = glob(["**/*.py"], allow_empty = True),
- data = glob(["**/*"], exclude={data_exclude}),
- # This makes this directory a top-level in the python import
- # search path for anything that depends on this.
- imports = ["."],
- deps = [{dependencies}],
- )
- """.format(
- name=name,
- dependencies=",".join(dependencies),
- data_exclude=json.dumps(data_exclude),
- whl_file_label=WHEEL_FILE_LABEL,
- whl_file_deps=",".join(whl_file_deps),
- )
- )
-
-
-def generate_requirements_file_contents(repo_name: str, targets: Iterable[str]) -> str:
- """Generate a requirements.bzl file for a given pip repository
-
- The file allows converting the PyPI name to a bazel label. Additionally, it adds a function which can glob all the
- installed dependencies.
-
- Args:
- repo_name: the name of the pip repository
- targets: a list of Bazel labels pointing to all the generated targets
-
- Returns:
- A complete requirements.bzl file as a string
- """
-
- sorted_targets = sorted(targets)
- requirement_labels = ",".join(sorted_targets)
- whl_requirement_labels = ",".join(
- '"{}:whl"'.format(target.strip('"')) for target in sorted_targets
- )
- return textwrap.dedent(
- """\
- all_requirements = [{requirement_labels}]
-
- all_whl_requirements = [{whl_requirement_labels}]
-
- def requirement(name):
- name_key = name.replace("-", "_").replace(".", "_").lower()
- return "{repo}//pypi__" + name_key
-
- def whl_requirement(name):
- return requirement(name) + ":whl"
-
- def install_deps():
- fail("install_deps() only works if you are creating an incremental repo. Did you mean to use pip_parse()?")
- """.format(
- repo=repo_name,
- requirement_labels=requirement_labels,
- whl_requirement_labels=whl_requirement_labels,
- )
- )
-
-
-DEFAULT_PACKAGE_PREFIX = "pypi__"
-
-
-def whl_library_repo_prefix(parent_repo: str) -> str:
- return "{parent}_{default_package_prefix}".format(
- parent=parent_repo,
- default_package_prefix=DEFAULT_PACKAGE_PREFIX
- )
-
-
-def sanitise_name(name: str, prefix: str = DEFAULT_PACKAGE_PREFIX) -> str:
- """Sanitises the name to be compatible with Bazel labels.
-
- There are certain requirements around Bazel labels that we need to consider. From the Bazel docs:
-
- Package names must be composed entirely of characters drawn from the set A-Z, a–z, 0–9, '/', '-', '.', and '_',
- and cannot start with a slash.
-
- Due to restrictions on Bazel labels we also cannot allow hyphens. See
- https://github.com/bazelbuild/bazel/issues/6841
-
- Further, rules-python automatically adds the repository root to the PYTHONPATH, meaning a package that has the same
- name as a module is picked up. We workaround this by prefixing with `pypi__`. Alternatively we could require
- `--noexperimental_python_import_all_repositories` be set, however this breaks rules_docker.
- See: https://github.com/bazelbuild/bazel/issues/2636
- """
-
- return prefix + name.replace("-", "_").replace(".", "_").lower()
-
-
-def setup_namespace_pkg_compatibility(wheel_dir: str) -> None:
- """Converts native namespace packages to pkgutil-style packages
-
- Namespace packages can be created in one of three ways. They are detailed here:
- https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package
-
- 'pkgutil-style namespace packages' (2) and 'pkg_resources-style namespace packages' (3) works in Bazel, but
- 'native namespace packages' (1) do not.
-
- We ensure compatibility with Bazel of method 1 by converting them into method 2.
-
- Args:
- wheel_dir: the directory of the wheel to convert
- """
-
- namespace_pkg_dirs = namespace_pkgs.implicit_namespace_packages(
- wheel_dir, ignored_dirnames=["%s/bin" % wheel_dir],
- )
-
- for ns_pkg_dir in namespace_pkg_dirs:
- namespace_pkgs.add_pkgutil_style_namespace_pkg_init(ns_pkg_dir)
-
-
-def sanitised_library_label(whl_name: str) -> str:
- return '"//%s"' % sanitise_name(whl_name)
-
-
-def sanitised_file_label(whl_name: str) -> str:
- return '"//%s:%s"' % (sanitise_name(whl_name), WHEEL_FILE_LABEL)
-
-
-def _whl_name_to_repo_root(whl_name: str, repo_prefix: str) -> str:
- return "@{}//".format(sanitise_name(whl_name, prefix=repo_prefix))
-
-
-def sanitised_repo_library_label(whl_name: str, repo_prefix: str) -> str:
- return '"{}:{}"'.format(_whl_name_to_repo_root(whl_name, repo_prefix), PY_LIBRARY_LABEL)
-
-
-def sanitised_repo_file_label(whl_name: str, repo_prefix: str) -> str:
- return '"{}:{}"'.format(_whl_name_to_repo_root(whl_name, repo_prefix), WHEEL_FILE_LABEL)
-
-
-def extract_wheel(
- wheel_file: str,
- extras: Dict[str, Set[str]],
- pip_data_exclude: List[str],
- enable_implicit_namespace_pkgs: bool,
- incremental: bool = False,
- incremental_repo_prefix: Optional[str] = None,
-) -> str:
- """Extracts wheel into given directory and creates py_library and filegroup targets.
-
- Args:
- wheel_file: the filepath of the .whl
- extras: a list of extras to add as dependencies for the installed wheel
- pip_data_exclude: list of file patterns to exclude from the generated data section of the py_library
- enable_implicit_namespace_pkgs: if true, disables conversion of implicit namespace packages and will unzip as-is
- incremental: If true the extract the wheel in a format suitable for an external repository. This
- effects the names of libraries and their dependencies, which point to other external repositories.
- incremental_repo_prefix: If incremental is true, use this prefix when creating labels from wheel
- names instead of the default.
-
- Returns:
- The Bazel label for the extracted wheel, in the form '//path/to/wheel'.
- """
-
- whl = wheel.Wheel(wheel_file)
- if incremental:
- directory = "."
- else:
- directory = sanitise_name(whl.name)
-
- os.mkdir(directory)
- # copy the original wheel
- shutil.copy(whl.path, directory)
- whl.unzip(directory)
-
- # Note: Order of operations matters here
- purelib.spread_purelib_into_root(directory)
-
- if not enable_implicit_namespace_pkgs:
- setup_namespace_pkg_compatibility(directory)
-
- extras_requested = extras[whl.name] if whl.name in extras else set()
- whl_deps = sorted(whl.dependencies(extras_requested))
-
- if incremental:
- # check for mypy Optional validity
- if incremental_repo_prefix is None:
- raise TypeError("incremental_repo_prefix arguement cannot be None if incremental == True")
- sanitised_dependencies = [
- sanitised_repo_library_label(d, repo_prefix=incremental_repo_prefix) for d in whl_deps
- ]
- sanitised_wheel_file_dependencies = [
- sanitised_repo_file_label(d, repo_prefix=incremental_repo_prefix) for d in whl_deps
- ]
- else:
- sanitised_dependencies = [
- sanitised_library_label(d) for d in whl_deps
- ]
- sanitised_wheel_file_dependencies = [
- sanitised_file_label(d) for d in whl_deps
- ]
-
- with open(os.path.join(directory, "BUILD.bazel"), "w") as build_file:
- contents = generate_build_file_contents(
- PY_LIBRARY_LABEL if incremental else sanitise_name(whl.name),
- sanitised_dependencies,
- sanitised_wheel_file_dependencies,
- pip_data_exclude
- )
- build_file.write(contents)
-
- if not incremental:
- os.remove(whl.path)
-
- return "//%s" % directory
diff --git a/python/pip_install/extract_wheels/lib/namespace_pkgs.py b/python/pip_install/extract_wheels/lib/namespace_pkgs.py
deleted file mode 100644
index ca5ffb5b13..0000000000
--- a/python/pip_install/extract_wheels/lib/namespace_pkgs.py
+++ /dev/null
@@ -1,70 +0,0 @@
-"""Utility functions to discover python package types"""
-import os
-import textwrap
-from typing import Set, List, Optional
-
-
-def implicit_namespace_packages(
- directory: str, ignored_dirnames: Optional[List[str]] = None
-) -> Set[str]:
- """Discovers namespace packages implemented using the 'native namespace packages' method.
-
- AKA 'implicit namespace packages', which has been supported since Python 3.3.
- See: https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages
-
- Args:
- directory: The root directory to recursively find packages in.
- ignored_dirnames: A list of directories to exclude from the search
-
- Returns:
- The set of directories found under root to be packages using the native namespace method.
- """
- namespace_pkg_dirs = set()
- for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
- # We are only interested in dirs with no __init__.py file
- if "__init__.py" in filenames:
- dirnames[:] = [] # Remove dirnames from search
- continue
-
- for ignored_dir in ignored_dirnames or []:
- if ignored_dir in dirnames:
- dirnames.remove(ignored_dir)
-
- non_empty_directory = dirnames or filenames
- if (
- non_empty_directory
- and
- # The root of the directory should never be an implicit namespace
- dirpath != directory
- ):
- namespace_pkg_dirs.add(dirpath)
-
- return namespace_pkg_dirs
-
-
-def add_pkgutil_style_namespace_pkg_init(dir_path: str) -> None:
- """Adds 'pkgutil-style namespace packages' init file to the given directory
-
- See: https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages
-
- Args:
- dir_path: The directory to create an __init__.py for.
-
- Raises:
- ValueError: If the directory already contains an __init__.py file
- """
- ns_pkg_init_filepath = os.path.join(dir_path, "__init__.py")
-
- if os.path.isfile(ns_pkg_init_filepath):
- raise ValueError("%s already contains an __init__.py file." % dir_path)
-
- with open(ns_pkg_init_filepath, "w") as ns_pkg_init_f:
- # See https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages
- ns_pkg_init_f.write(
- textwrap.dedent(
- """\
- # __path__ manipulation added by rules_python_external to support namespace pkgs.
- __path__ = __import__('pkgutil').extend_path(__path__, __name__)
- """
- )
- )
diff --git a/python/pip_install/extract_wheels/lib/namespace_pkgs_test.py b/python/pip_install/extract_wheels/lib/namespace_pkgs_test.py
deleted file mode 100644
index 5eec5c3199..0000000000
--- a/python/pip_install/extract_wheels/lib/namespace_pkgs_test.py
+++ /dev/null
@@ -1,73 +0,0 @@
-import pathlib
-import shutil
-import tempfile
-from typing import Optional
-import unittest
-
-from python.pip_install.extract_wheels.lib import namespace_pkgs
-
-
-class TempDir:
- def __init__(self) -> None:
- self.dir = tempfile.mkdtemp()
-
- def root(self) -> str:
- return self.dir
-
- def add_dir(self, rel_path: str) -> None:
- d = pathlib.Path(self.dir, rel_path)
- d.mkdir(parents=True)
-
- def add_file(self, rel_path: str, contents: Optional[str] = None) -> None:
- f = pathlib.Path(self.dir, rel_path)
- f.parent.mkdir(parents=True, exist_ok=True)
- if contents:
- with open(str(f), "w") as writeable_f:
- writeable_f.write(contents)
- else:
- f.touch()
-
- def remove(self) -> None:
- shutil.rmtree(self.dir)
-
-
-class TestImplicitNamespacePackages(unittest.TestCase):
- def test_finds_correct_namespace_packages(self) -> None:
- directory = TempDir()
- directory.add_file("foo/bar/biz.py")
- directory.add_file("foo/bee/boo.py")
- directory.add_file("foo/buu/__init__.py")
- directory.add_file("foo/buu/bii.py")
-
- expected = {
- directory.root() + "/foo",
- directory.root() + "/foo/bar",
- directory.root() + "/foo/bee",
- }
- actual = namespace_pkgs.implicit_namespace_packages(directory.root())
- self.assertEqual(actual, expected)
-
- def test_ignores_empty_directories(self) -> None:
- directory = TempDir()
- directory.add_file("foo/bar/biz.py")
- directory.add_dir("foo/cat")
-
- expected = {
- directory.root() + "/foo",
- directory.root() + "/foo/bar",
- }
- actual = namespace_pkgs.implicit_namespace_packages(directory.root())
- self.assertEqual(actual, expected)
-
- def test_empty_case(self) -> None:
- directory = TempDir()
- directory.add_file("foo/__init__.py")
- directory.add_file("foo/bar/__init__.py")
- directory.add_file("foo/bar/biz.py")
-
- actual = namespace_pkgs.implicit_namespace_packages(directory.root())
- self.assertEqual(actual, set())
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/extract_wheels/lib/purelib.py b/python/pip_install/extract_wheels/lib/purelib.py
deleted file mode 100644
index 4e9eb3f3ef..0000000000
--- a/python/pip_install/extract_wheels/lib/purelib.py
+++ /dev/null
@@ -1,56 +0,0 @@
-"""Functions to make purelibs Bazel compatible"""
-import pathlib
-import shutil
-
-from python.pip_install.extract_wheels.lib import wheel
-
-
-def spread_purelib_into_root(wheel_dir: str) -> None:
- """Unpacks purelib directories into the root.
-
- Args:
- wheel_dir: The root of the extracted wheel directory.
- """
- dist_info = wheel.get_dist_info(wheel_dir)
- wheel_metadata_file_path = pathlib.Path(dist_info, "WHEEL")
- wheel_metadata_dict = wheel.parse_wheel_meta_file(str(wheel_metadata_file_path))
-
- if "Root-Is-Purelib" not in wheel_metadata_dict:
- raise ValueError(
- "Invalid WHEEL file '%s'. Expected key 'Root-Is-Purelib'."
- % wheel_metadata_file_path
- )
- root_is_purelib = wheel_metadata_dict["Root-Is-Purelib"]
-
- if root_is_purelib.lower() == "true":
- # The Python package code is in the root of the Wheel, so no need to 'spread' anything.
- return
-
- dot_data_dir = wheel.get_dot_data_directory(wheel_dir)
- # 'Root-Is-Purelib: false' is no guarantee a .data directory exists with
- # package code in it. eg. the 'markupsafe' package.
- if not dot_data_dir:
- return
-
- for child in pathlib.Path(dot_data_dir).iterdir():
- # TODO(Jonathon): Should all other potential folders get ignored? eg. 'platlib'
- if str(child).endswith("purelib"):
- _spread_purelib(child, wheel_dir)
-
-
-def _spread_purelib(purelib_dir: pathlib.Path, root_dir: str) -> None:
- """Recursively moves all sibling directories of the purelib to the root.
-
- Args:
- purelib_dir: The directory of the purelib.
- root_dir: The directory to move files into.
- """
- for grandchild in purelib_dir.iterdir():
- # Some purelib Wheels, like Tensorflow 2.0.0, have directories
- # split between the root and the purelib directory. In this case
- # we should leave the purelib 'sibling' alone.
- # See: https://github.com/dillon-giacoppo/rules_python_external/issues/8
- if not pathlib.Path(root_dir, grandchild.name).exists():
- shutil.move(
- src=str(grandchild), dst=root_dir,
- )
diff --git a/python/pip_install/extract_wheels/lib/requirements.py b/python/pip_install/extract_wheels/lib/requirements.py
deleted file mode 100644
index e246379bcc..0000000000
--- a/python/pip_install/extract_wheels/lib/requirements.py
+++ /dev/null
@@ -1,45 +0,0 @@
-import re
-from typing import Dict, Set, Tuple, Optional
-
-
-def parse_extras(requirements_path: str) -> Dict[str, Set[str]]:
- """Parse over the requirements.txt file to find extras requested.
-
- Args:
- requirements_path: The filepath for the requirements.txt file to parse.
-
- Returns:
- A dictionary mapping the requirement name to a set of extras requested.
- """
-
- extras_requested = {}
- with open(requirements_path, "r") as requirements:
- # Merge all backslash line continuations so we parse each requirement as a single line.
- for line in requirements.read().replace("\\\n", "").split("\n"):
- requirement, extras = _parse_requirement_for_extra(line)
- if requirement and extras:
- extras_requested[requirement] = extras
-
- return extras_requested
-
-
-def _parse_requirement_for_extra(
- requirement: str,
-) -> Tuple[Optional[str], Optional[Set[str]]]:
- """Given a requirement string, returns the requirement name and set of extras, if extras specified.
- Else, returns (None, None)
- """
-
- # https://www.python.org/dev/peps/pep-0508/#grammar
- extras_pattern = re.compile(
- r"^\s*([0-9A-Za-z][0-9A-Za-z_.\-]*)\s*\[\s*([0-9A-Za-z][0-9A-Za-z_.\-]*(?:\s*,\s*[0-9A-Za-z][0-9A-Za-z_.\-]*)*)\s*\]"
- )
-
- matches = extras_pattern.match(requirement)
- if matches:
- return (
- matches.group(1),
- {extra.strip() for extra in matches.group(2).split(",")},
- )
-
- return None, None
diff --git a/python/pip_install/extract_wheels/lib/requirements_bzl_test.py b/python/pip_install/extract_wheels/lib/requirements_bzl_test.py
deleted file mode 100644
index 3424f3e9b7..0000000000
--- a/python/pip_install/extract_wheels/lib/requirements_bzl_test.py
+++ /dev/null
@@ -1,17 +0,0 @@
-import unittest
-
-from python.pip_install.extract_wheels.lib import bazel
-
-
-class TestGenerateRequirementsFileContents(unittest.TestCase):
- def test_all_wheel_requirements(self) -> None:
- contents = bazel.generate_requirements_file_contents(
- repo_name='test',
- targets=['"@test//pypi__pkg1"', '"@test//pypi__pkg2"'],
- )
- expected = 'all_whl_requirements = ["@test//pypi__pkg1:whl","@test//pypi__pkg2:whl"]'
- self.assertIn(expected, contents)
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/extract_wheels/lib/requirements_test.py b/python/pip_install/extract_wheels/lib/requirements_test.py
deleted file mode 100644
index ba7ee13a69..0000000000
--- a/python/pip_install/extract_wheels/lib/requirements_test.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import unittest
-
-from python.pip_install.extract_wheels.lib import requirements
-
-
-class TestRequirementExtrasParsing(unittest.TestCase):
- def test_parses_requirement_for_extra(self) -> None:
- cases = [
- ("name[foo]", ("name", frozenset(["foo"]))),
- ("name[ Foo123 ]", ("name", frozenset(["Foo123"]))),
- (" name1[ foo ] ", ("name1", frozenset(["foo"]))),
- (
- "name [fred,bar] @ http://foo.com ; python_version=='2.7'",
- ("name", frozenset(["fred", "bar"])),
- ),
- (
- "name[quux, strange];python_version<'2.7' and platform_version=='2'",
- ("name", frozenset(["quux", "strange"])),
- ),
- ("name; (os_name=='a' or os_name=='b') and os_name=='c'", (None, None),),
- ("name@http://foo.com", (None, None),),
- ]
-
- for case, expected in cases:
- with self.subTest():
- self.assertTupleEqual(
- requirements._parse_requirement_for_extra(case), expected
- )
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/extract_wheels/lib/wheel.py b/python/pip_install/extract_wheels/lib/wheel.py
deleted file mode 100644
index c13f4e8621..0000000000
--- a/python/pip_install/extract_wheels/lib/wheel.py
+++ /dev/null
@@ -1,147 +0,0 @@
-"""Utility class to inspect an extracted wheel directory"""
-import glob
-import os
-import stat
-import zipfile
-from typing import Dict, Optional, Set
-
-import pkg_resources
-import pkginfo
-
-
-def current_umask() -> int:
- """Get the current umask which involves having to set it temporarily."""
- mask = os.umask(0)
- os.umask(mask)
- return mask
-
-
-def set_extracted_file_to_default_mode_plus_executable(path: str) -> None:
- """
- Make file present at path have execute for user/group/world
- (chmod +x) is no-op on windows per python docs
- """
- os.chmod(path, (0o777 & ~current_umask() | 0o111))
-
-
-class Wheel:
- """Representation of the compressed .whl file"""
-
- def __init__(self, path: str):
- self._path = path
-
- @property
- def path(self) -> str:
- return self._path
-
- @property
- def name(self) -> str:
- return str(self.metadata.name)
-
- @property
- def metadata(self) -> pkginfo.Wheel:
- return pkginfo.get_metadata(self.path)
-
- def dependencies(self, extras_requested: Optional[Set[str]] = None) -> Set[str]:
- dependency_set = set()
-
- for wheel_req in self.metadata.requires_dist:
- req = pkg_resources.Requirement(wheel_req) # type: ignore
-
- if req.marker is None or any(
- req.marker.evaluate({"extra": extra})
- for extra in extras_requested or [""]
- ):
- dependency_set.add(req.name) # type: ignore
-
- return dependency_set
-
- def unzip(self, directory: str) -> None:
- with zipfile.ZipFile(self.path, "r") as whl:
- whl.extractall(directory)
- # The following logic is borrowed from Pip:
- # https://github.com/pypa/pip/blob/cc48c07b64f338ac5e347d90f6cb4efc22ed0d0b/src/pip/_internal/utils/unpacking.py#L240
- for info in whl.infolist():
- name = info.filename
- # Do not attempt to modify directories.
- if name.endswith("/") or name.endswith("\\"):
- continue
- mode = info.external_attr >> 16
- # if mode and regular file and any execute permissions for
- # user/group/world?
- if mode and stat.S_ISREG(mode) and mode & 0o111:
- name = os.path.join(directory, name)
- set_extracted_file_to_default_mode_plus_executable(name)
-
-
-def get_dist_info(wheel_dir: str) -> str:
- """"Returns the relative path to the dist-info directory if it exists.
-
- Args:
- wheel_dir: The root of the extracted wheel directory.
-
- Returns:
- Relative path to the dist-info directory if it exists, else, None.
- """
- dist_info_dirs = glob.glob(os.path.join(wheel_dir, "*.dist-info"))
- if not dist_info_dirs:
- raise ValueError(
- "No *.dist-info directory found. %s is not a valid Wheel." % wheel_dir
- )
-
- if len(dist_info_dirs) > 1:
- raise ValueError(
- "Found more than 1 *.dist-info directory. %s is not a valid Wheel."
- % wheel_dir
- )
-
- return dist_info_dirs[0]
-
-
-def get_dot_data_directory(wheel_dir: str) -> Optional[str]:
- """Returns the relative path to the data directory if it exists.
-
- See: https://www.python.org/dev/peps/pep-0491/#the-data-directory
-
- Args:
- wheel_dir: The root of the extracted wheel directory.
-
- Returns:
- Relative path to the data directory if it exists, else, None.
- """
-
- dot_data_dirs = glob.glob(os.path.join(wheel_dir, "*.data"))
- if not dot_data_dirs:
- return None
-
- if len(dot_data_dirs) > 1:
- raise ValueError(
- "Found more than 1 *.data directory. %s is not a valid Wheel." % wheel_dir
- )
-
- return dot_data_dirs[0]
-
-
-def parse_wheel_meta_file(wheel_dir: str) -> Dict[str, str]:
- """Parses the given WHEEL file into a dictionary.
-
- Args:
- wheel_dir: The file path of the WHEEL metadata file in dist-info.
-
- Returns:
- The WHEEL file mapped into a dictionary.
- """
- contents = {}
- with open(wheel_dir, "r") as wheel_file:
- for line in wheel_file:
- cleaned = line.strip()
- if not cleaned:
- continue
- try:
- key, value = cleaned.split(":", maxsplit=1)
- contents[key] = value.strip()
- except ValueError:
- raise RuntimeError(
- "Encounted invalid line in WHEEL file: '%s'" % cleaned
- )
- return contents
diff --git a/python/pip_install/extract_wheels/lib/whl_filegroup_test.py b/python/pip_install/extract_wheels/lib/whl_filegroup_test.py
deleted file mode 100644
index 84054b1725..0000000000
--- a/python/pip_install/extract_wheels/lib/whl_filegroup_test.py
+++ /dev/null
@@ -1,52 +0,0 @@
-import os
-import shutil
-import tempfile
-from typing import Optional
-import unittest
-
-from python.pip_install.extract_wheels.lib import bazel
-
-
-class TestWhlFilegroup(unittest.TestCase):
- def setUp(self) -> None:
- self.wheel_name = "example_minimal_package-0.0.1-py3-none-any.whl"
- self.wheel_dir = tempfile.mkdtemp()
- self.wheel_path = os.path.join(self.wheel_dir, self.wheel_name)
- shutil.copy(
- os.path.join("examples", "wheel", self.wheel_name), self.wheel_dir
- )
- self.original_dir = os.getcwd()
- os.chdir(self.wheel_dir)
-
- def tearDown(self):
- shutil.rmtree(self.wheel_dir)
- os.chdir(self.original_dir)
-
- def _run(
- self,
- incremental: bool = False,
- incremental_repo_prefix: Optional[str] = None,
- ) -> None:
- generated_bazel_dir = bazel.extract_wheel(
- self.wheel_path,
- extras={},
- pip_data_exclude=[],
- enable_implicit_namespace_pkgs=False,
- incremental=incremental,
- incremental_repo_prefix=incremental_repo_prefix
- )[2:] # Take off the leading // from the returned label.
- # Assert that the raw wheel ends up in the package.
- self.assertIn(self.wheel_name, os.listdir(generated_bazel_dir))
- with open("{}/BUILD.bazel".format(generated_bazel_dir)) as build_file:
- build_file_content = build_file.read()
- self.assertIn('filegroup', build_file_content)
-
- def test_nonincremental(self) -> None:
- self._run()
-
- def test_incremental(self) -> None:
- self._run(incremental=True, incremental_repo_prefix="test")
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/parse_requirements_to_bzl/BUILD b/python/pip_install/parse_requirements_to_bzl/BUILD
deleted file mode 100644
index bb6032382f..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/BUILD
+++ /dev/null
@@ -1,43 +0,0 @@
-load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
-load("//python/pip_install:repositories.bzl", "requirement")
-
-py_binary(
- name = "parse_requirements_to_bzl",
- srcs = [
- "__init__.py",
- "__main__.py",
- ],
- main = "__main__.py",
- deps = ["//python/pip_install/extract_wheels/lib"],
-)
-
-py_library(
- name = "lib",
- srcs = ["__init__.py"],
- visibility = ["//python/pip_install/extract_wheels:__subpackages__"],
- deps = [requirement("pip")],
-)
-
-py_test(
- name = "parse_requirements_to_bzl_test",
- size = "small",
- srcs = [
- "parse_requirements_to_bzl_test.py",
- ],
- tags = ["unit"],
- deps = [
- ":lib",
- "//python/pip_install/extract_wheels/lib",
- ],
-)
-
-filegroup(
- name = "distribution",
- srcs = glob(
- ["*"],
- exclude = ["*_test.py"],
- ) + [
- "//python/pip_install/parse_requirements_to_bzl/extract_single_wheel:distribution",
- ],
- visibility = ["//python/pip_install:__subpackages__"],
-)
diff --git a/python/pip_install/parse_requirements_to_bzl/__init__.py b/python/pip_install/parse_requirements_to_bzl/__init__.py
deleted file mode 100644
index 66e6f5e817..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/__init__.py
+++ /dev/null
@@ -1,153 +0,0 @@
-import argparse
-import json
-import textwrap
-import sys
-import shlex
-from typing import List, Tuple
-
-from python.pip_install.extract_wheels.lib import bazel, arguments
-from pip._internal.req import parse_requirements, constructors
-from pip._internal.req.req_install import InstallRequirement
-from pip._internal.req.req_file import get_file_content, preprocess, handle_line, get_line_parser, RequirementsFileParser
-from pip._internal.network.session import PipSession
-
-
-def parse_install_requirements(requirements_lock: str, extra_pip_args: List[str]) -> List[Tuple[InstallRequirement, str]]:
- ps = PipSession()
- # This is roughly taken from pip._internal.req.req_file.parse_requirements
- # (https://github.com/pypa/pip/blob/21.0.1/src/pip/_internal/req/req_file.py#L127) in order to keep
- # the original line (sort-of, its preprocessed) from the requirements_lock file around, to pass to sub repos
- # as the requirement.
- line_parser = get_line_parser(finder=None)
- parser = RequirementsFileParser(ps, line_parser)
- install_req_and_lines: List[Tuple[InstallRequirement, str]] = []
- _, content = get_file_content(requirements_lock, ps)
- for parsed_line, (_, line) in zip(parser.parse(requirements_lock, constraint=False), preprocess(content)):
- if parsed_line.is_requirement:
- install_req_and_lines.append(
- (
- constructors.install_req_from_line(parsed_line.requirement),
- line
- )
- )
-
- else:
- extra_pip_args.extend(shlex.split(line))
- return install_req_and_lines
-
-
-def repo_names_and_requirements(install_reqs: List[Tuple[InstallRequirement, str]], repo_prefix: str) -> List[Tuple[str, str]]:
- return [
- (
- bazel.sanitise_name(ir.name, prefix=repo_prefix),
- line,
- )
- for ir, line in install_reqs
- ]
-
-def deserialize_structured_args(args):
- """Deserialize structured arguments passed from the starlark rules.
- Args:
- args: dict of parsed command line arguments
- """
- structured_args = ("extra_pip_args", "pip_data_exclude")
- for arg_name in structured_args:
- if args.get(arg_name) is not None:
- args[arg_name] = json.loads(args[arg_name])["args"]
- else:
- args[arg_name] = []
- return args
-
-
-def generate_parsed_requirements_contents(all_args: argparse.Namespace) -> str:
- """
- Parse each requirement from the requirements_lock file, and prepare arguments for each
- repository rule, which will represent the individual requirements.
-
- Generates a requirements.bzl file containing a macro (install_deps()) which instantiates
- a repository rule for each requirment in the lock file.
- """
-
- args = dict(vars(all_args))
- args = deserialize_structured_args(args)
- args.setdefault("python_interpreter", sys.executable)
- # Pop this off because it wont be used as a config argument to the whl_library rule.
- requirements_lock = args.pop("requirements_lock")
- repo_prefix = bazel.whl_library_repo_prefix(args["repo"])
-
- install_req_and_lines = parse_install_requirements(requirements_lock, args["extra_pip_args"])
- repo_names_and_reqs = repo_names_and_requirements(install_req_and_lines, repo_prefix)
- all_requirements = ", ".join(
- [bazel.sanitised_repo_library_label(ir.name, repo_prefix=repo_prefix) for ir, _ in install_req_and_lines]
- )
- all_whl_requirements = ", ".join(
- [bazel.sanitised_repo_file_label(ir.name, repo_prefix=repo_prefix) for ir, _ in install_req_and_lines]
- )
- return textwrap.dedent("""\
- load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")
-
- all_requirements = [{all_requirements}]
-
- all_whl_requirements = [{all_whl_requirements}]
-
- _packages = {repo_names_and_reqs}
- _config = {args}
-
- def _clean_name(name):
- return name.replace("-", "_").replace(".", "_").lower()
-
- def requirement(name):
- return "@{repo_prefix}" + _clean_name(name) + "//:pkg"
-
- def whl_requirement(name):
- return "@{repo_prefix}" + _clean_name(name) + "//:whl"
-
- def install_deps():
- for name, requirement in _packages:
- whl_library(
- name = name,
- requirement = requirement,
- **_config,
- )
- """.format(
- all_requirements=all_requirements,
- all_whl_requirements=all_whl_requirements,
- repo_names_and_reqs=repo_names_and_reqs,
- args=args,
- repo_prefix=repo_prefix,
- )
- )
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(
- description="Create rules to incrementally fetch needed \
-dependencies from a fully resolved requirements lock file."
- )
- parser.add_argument(
- "--requirements_lock",
- action="store",
- required=True,
- help="Path to fully resolved requirements.txt to use as the source of repos.",
- )
- parser.add_argument(
- "--quiet",
- type=bool,
- action="store",
- required=True,
- help="Whether to print stdout / stderr from child repos.",
- )
- parser.add_argument(
- "--timeout",
- type=int,
- action="store",
- required=True,
- help="timeout to use for pip operation.",
- )
- arguments.parse_common_args(parser)
- args = parser.parse_args()
-
- with open("requirements.bzl", "w") as requirement_file:
- requirement_file.write(
- generate_parsed_requirements_contents(args)
- )
diff --git a/python/pip_install/parse_requirements_to_bzl/__main__.py b/python/pip_install/parse_requirements_to_bzl/__main__.py
deleted file mode 100644
index 89199612b5..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/__main__.py
+++ /dev/null
@@ -1,5 +0,0 @@
-"""Main entry point."""
-from python.pip_install.parse_requirements_to_bzl import main
-
-if __name__ == "__main__":
- main()
diff --git a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/BUILD b/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/BUILD
deleted file mode 100644
index 17bdfe75ce..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/BUILD
+++ /dev/null
@@ -1,8 +0,0 @@
-filegroup(
- name = "distribution",
- srcs = glob(
- ["*"],
- exclude = ["*_test.py"],
- ),
- visibility = ["//python/pip_install:__subpackages__"],
-)
diff --git a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py b/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py
deleted file mode 100644
index 884b8ad575..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py
+++ /dev/null
@@ -1,58 +0,0 @@
-import argparse
-import sys
-import glob
-import subprocess
-import json
-
-from tempfile import NamedTemporaryFile
-
-from python.pip_install.extract_wheels.lib import bazel, requirements, arguments
-from python.pip_install.extract_wheels import configure_reproducible_wheels
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(
- description="Build and/or fetch a single wheel based on the requirement passed in"
- )
- parser.add_argument(
- "--requirement",
- action="store",
- required=True,
- help="A single PEP508 requirement specifier string.",
- )
- arguments.parse_common_args(parser)
- args = parser.parse_args()
-
- configure_reproducible_wheels()
-
- pip_args = [sys.executable, "-m", "pip", "--isolated", "wheel", "--no-deps"]
- if args.extra_pip_args:
- pip_args += json.loads(args.extra_pip_args)["args"]
-
- with NamedTemporaryFile(mode='wb') as requirement_file:
- requirement_file.write(args.requirement.encode("utf-8"))
- requirement_file.flush()
- # Requirement specific args like --hash can only be passed in a requirements file,
- # so write our single requirement into a temp file in case it has any of those flags.
- pip_args.extend(["-r", requirement_file.name])
-
- # Assumes any errors are logged by pip so do nothing. This command will fail if pip fails
- subprocess.run(pip_args, check=True)
-
- name, extras_for_pkg = requirements._parse_requirement_for_extra(args.requirement)
- extras = {name: extras_for_pkg} if extras_for_pkg and name else dict()
-
- if args.pip_data_exclude:
- pip_data_exclude = json.loads(args.pip_data_exclude)["exclude"]
- else:
- pip_data_exclude = []
-
- whl = next(iter(glob.glob("*.whl")))
- bazel.extract_wheel(
- whl,
- extras,
- pip_data_exclude,
- args.enable_implicit_namespace_pkgs,
- incremental=True,
- incremental_repo_prefix=bazel.whl_library_repo_prefix(args.repo)
- )
diff --git a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py b/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py
deleted file mode 100644
index d45f90bbd1..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py
+++ /dev/null
@@ -1,4 +0,0 @@
-from python.pip_install.parse_requirements_to_bzl.extract_single_wheel import main
-
-if __name__ == "__main__":
- main()
diff --git a/python/pip_install/parse_requirements_to_bzl/parse_requirements_to_bzl_test.py b/python/pip_install/parse_requirements_to_bzl/parse_requirements_to_bzl_test.py
deleted file mode 100644
index 7199cea0cc..0000000000
--- a/python/pip_install/parse_requirements_to_bzl/parse_requirements_to_bzl_test.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import unittest
-import argparse
-import json
-from tempfile import NamedTemporaryFile
-
-from python.pip_install.parse_requirements_to_bzl import generate_parsed_requirements_contents
-from python.pip_install.extract_wheels.lib.bazel import (
- sanitised_repo_library_label,
- whl_library_repo_prefix,
- sanitised_repo_file_label
-)
-
-
-class TestParseRequirementsToBzl(unittest.TestCase):
-
- def test_generated_requirements_bzl(self) -> None:
- with NamedTemporaryFile() as requirements_lock:
- comments_and_flags = "#comment\n--require-hashes True\n"
- requirement_string = "foo==0.0.0 --hash=sha256:hashofFoowhl"
- requirements_lock.write(bytes(comments_and_flags + requirement_string, encoding="utf-8"))
- requirements_lock.flush()
- args = argparse.Namespace()
- args.requirements_lock = requirements_lock.name
- args.repo = "pip_parsed_deps"
- extra_pip_args = ["--index-url=pypi.org/simple"]
- args.extra_pip_args = json.dumps({"args": extra_pip_args})
- contents = generate_parsed_requirements_contents(args)
- library_target = "@pip_parsed_deps_pypi__foo//:pkg"
- whl_target = "@pip_parsed_deps_pypi__foo//:whl"
- all_requirements = 'all_requirements = ["{library_target}"]'.format(library_target=library_target)
- all_whl_requirements = 'all_whl_requirements = ["{whl_target}"]'.format(whl_target=whl_target)
- self.assertIn(all_requirements, contents, contents)
- self.assertIn(all_whl_requirements, contents, contents)
- self.assertIn(requirement_string, contents, contents)
- self.assertIn(requirement_string, contents, contents)
- all_flags = extra_pip_args + ["--require-hashes", "True"]
- self.assertIn("'extra_pip_args': {}".format(repr(all_flags)), contents, contents)
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/python/pip_install/pip_compile.py b/python/pip_install/pip_compile.py
deleted file mode 100644
index 1c22d2c0af..0000000000
--- a/python/pip_install/pip_compile.py
+++ /dev/null
@@ -1,89 +0,0 @@
-"Set defaults for the pip-compile command to run it under Bazel"
-
-import os
-import sys
-from shutil import copyfile
-
-from piptools.scripts.compile import cli
-
-if len(sys.argv) < 4:
- print(
- "Expected at least two arguments: requirements_in requirements_out",
- file=sys.stderr,
- )
- sys.exit(1)
-
-requirements_in = os.path.relpath(sys.argv.pop(1))
-requirements_txt = sys.argv.pop(1)
-update_target_name = sys.argv.pop(1)
-
-# Before loading click, set the locale for its parser.
-# If it leaks through to the system setting, it may fail:
-# RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII
-# as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for
-# mitigation steps.
-os.environ["LC_ALL"] = "C.UTF-8"
-os.environ["LANG"] = "C.UTF-8"
-
-UPDATE = True
-# Detect if we are running under `bazel test`
-if "TEST_TMPDIR" in os.environ:
- UPDATE = False
- # pip-compile wants the cache files to be writeable, but if we point
- # to the real user cache, Bazel sandboxing makes the file read-only
- # and we fail.
- # In theory this makes the test more hermetic as well.
- sys.argv.append("--cache-dir")
- sys.argv.append(os.environ["TEST_TMPDIR"])
- # Make a copy for pip-compile to read and mutate
- requirements_out = os.path.join(
- os.environ["TEST_TMPDIR"], os.path.basename(requirements_txt) + ".out"
- )
- copyfile(requirements_txt, requirements_out)
-
-elif "BUILD_WORKING_DIRECTORY" in os.environ:
- os.chdir(os.environ['BUILD_WORKING_DIRECTORY'])
-else:
- print(
- "Expected to find BUILD_WORKING_DIRECTORY in environment",
- file=sys.stderr,
- )
- sys.exit(1)
-
-update_target_pkg = "/".join(requirements_in.split('/')[:-1])
-# $(rootpath) in the workspace root gives ./requirements.in
-if update_target_pkg == ".":
- update_target_pkg = ""
-update_command = os.getenv("CUSTOM_COMPILE_COMMAND") or "bazel run //%s:%s" % (update_target_pkg, update_target_name)
-
-os.environ["CUSTOM_COMPILE_COMMAND"] = update_command
-
-sys.argv.append("--generate-hashes")
-sys.argv.append("--output-file")
-sys.argv.append(requirements_txt if UPDATE else requirements_out)
-sys.argv.append(requirements_in)
-
-if UPDATE:
- print("Updating " + requirements_txt)
- cli()
-else:
- # cli will exit(0) on success
- try:
- print("Checking " + requirements_txt)
- cli()
- print("cli() should exit", file=sys.stderr)
- sys.exit(1)
- except SystemExit:
- golden = open(requirements_txt).readlines()
- out = open(requirements_out).readlines()
- if golden != out:
- import difflib
-
- print(''.join(difflib.unified_diff(golden, out)), file=sys.stderr)
- print(
- "Lock file out of date. Run '"
- + update_command
- + "' to update.",
- file=sys.stderr,
- )
- sys.exit(1)
diff --git a/python/pip_install/pip_repository.bzl b/python/pip_install/pip_repository.bzl
index da4678f916..18deee1993 100644
--- a/python/pip_install/pip_repository.bzl
+++ b/python/pip_install/pip_repository.bzl
@@ -1,269 +1,26 @@
-""
-
-load("//python/pip_install:repositories.bzl", "all_requirements")
-
-def _construct_pypath(rctx):
- """Helper function to construct a PYTHONPATH.
-
- Contains entries for code in this repo as well as packages downloaded from //python/pip_install:repositories.bzl.
- This allows us to run python code inside repository rule implementations.
-
- Args:
- rctx: Handle to the repository_context.
- Returns: String of the PYTHONPATH.
- """
-
- # Get the root directory of these rules
- rules_root = rctx.path(Label("//:BUILD")).dirname
- thirdparty_roots = [
- # Includes all the external dependencies from repositories.bzl
- rctx.path(Label("@" + repo + "//:BUILD.bazel")).dirname
- for repo in all_requirements
- ]
- separator = ":" if not "windows" in rctx.os.name.lower() else ";"
- pypath = separator.join([str(p) for p in [rules_root] + thirdparty_roots])
- return pypath
-
-def _parse_optional_attrs(rctx, args):
- """Helper function to parse common attributes of pip_repository and whl_library repository rules.
-
- Args:
- rctx: Handle to the rule repository context.
- args: A list of parsed args for the rule.
- Returns: Augmented args list.
- """
- if rctx.attr.extra_pip_args:
- args += [
- "--extra_pip_args",
- struct(args = rctx.attr.extra_pip_args).to_json(),
- ]
-
- if rctx.attr.pip_data_exclude:
- args += [
- "--pip_data_exclude",
- struct(exclude = rctx.attr.pip_data_exclude).to_json(),
- ]
-
- if rctx.attr.enable_implicit_namespace_pkgs:
- args.append("--enable_implicit_namespace_pkgs")
-
- return args
-
-_BUILD_FILE_CONTENTS = """\
-package(default_visibility = ["//visibility:public"])
-
-# Ensure the `requirements.bzl` source can be accessed by stardoc, since users load() from it
-exports_files(["requirements.bzl"])
-"""
-
-def _pip_repository_impl(rctx):
- python_interpreter = rctx.attr.python_interpreter
- if rctx.attr.python_interpreter_target != None:
- target = rctx.attr.python_interpreter_target
- python_interpreter = rctx.path(target)
- else:
- if "/" not in python_interpreter:
- python_interpreter = rctx.which(python_interpreter)
- if not python_interpreter:
- fail("python interpreter not found")
-
- if rctx.attr.incremental and not rctx.attr.requirements_lock:
- fail("Incremental mode requires a requirements_lock attribute be specified.")
-
- # We need a BUILD file to load the generated requirements.bzl
- rctx.file("BUILD.bazel", _BUILD_FILE_CONTENTS)
-
- pypath = _construct_pypath(rctx)
-
- if rctx.attr.incremental:
- args = [
- python_interpreter,
- "-m",
- "python.pip_install.parse_requirements_to_bzl",
- "--requirements_lock",
- rctx.path(rctx.attr.requirements_lock),
- # pass quiet and timeout args through to child repos.
- "--quiet",
- str(rctx.attr.quiet),
- "--timeout",
- str(rctx.attr.timeout),
- ]
- else:
- args = [
- python_interpreter,
- "-m",
- "python.pip_install.extract_wheels",
- "--requirements",
- rctx.path(rctx.attr.requirements),
- ]
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
- args += ["--repo", rctx.attr.name]
- args = _parse_optional_attrs(rctx, args)
-
- result = rctx.execute(
- args,
- environment = {
- # Manually construct the PYTHONPATH since we cannot use the toolchain here
- "PYTHONPATH": pypath,
- },
- timeout = rctx.attr.timeout,
- quiet = rctx.attr.quiet,
- )
-
- if result.return_code:
- fail("rules_python failed: %s (%s)" % (result.stdout, result.stderr))
-
- return
-
-common_attrs = {
- "enable_implicit_namespace_pkgs": attr.bool(
- default = False,
- doc = """
-If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary
-and py_test targets must specify either `legacy_create_init=False` or the global Bazel option
-`--incompatible_default_to_explicit_init_py` to prevent `__init__.py` being automatically generated in every directory.
-
-This option is required to support some packages which cannot handle the conversion to pkg-util style.
- """,
- ),
- "extra_pip_args": attr.string_list(
- doc = "Extra arguments to pass on to pip. Must not contain spaces.",
- ),
- "pip_data_exclude": attr.string_list(
- doc = "Additional data exclusion parameters to add to the pip packages BUILD file.",
- ),
- "python_interpreter": attr.string(default = "python3"),
- "python_interpreter_target": attr.label(
- allow_single_file = True,
- doc = """
-If you are using a custom python interpreter built by another repository rule,
-use this attribute to specify its BUILD target. This allows pip_repository to invoke
-pip using the same interpreter as your toolchain. If set, takes precedence over
-python_interpreter.
-""",
- ),
- "quiet": attr.bool(
- default = True,
- doc = "If True, suppress printing stdout and stderr output to the terminal.",
- ),
- # 600 is documented as default here: https://docs.bazel.build/versions/master/skylark/lib/repository_ctx.html#execute
- "timeout": attr.int(
- default = 600,
- doc = "Timeout (in seconds) on the rule's execution duration.",
- ),
-}
-
-pip_repository_attrs = {
- "incremental": attr.bool(
- default = False,
- doc = "Create the repository in incremental mode.",
- ),
- "requirements": attr.label(
- allow_single_file = True,
- doc = "A 'requirements.txt' pip requirements file.",
- ),
- "requirements_lock": attr.label(
- allow_single_file = True,
- doc = """
-A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead
-of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that
-wheels are fetched/built only for the targets specified by 'build/run/test'.
-""",
- ),
-}
-
-pip_repository_attrs.update(**common_attrs)
-
-pip_repository = repository_rule(
- attrs = pip_repository_attrs,
- doc = """A rule for importing `requirements.txt` dependencies into Bazel.
-
-This rule imports a `requirements.txt` file and generates a new
-`requirements.bzl` file. This is used via the `WORKSPACE` pattern:
-
-```python
-pip_repository(
- name = "foo",
- requirements = ":requirements.txt",
-)
-```
-
-You can then reference imported dependencies from your `BUILD` file with:
-
-```python
-load("@foo//:requirements.bzl", "requirement")
-py_library(
- name = "bar",
- ...
- deps = [
- "//my/other:dep",
- requirement("requests"),
- requirement("numpy"),
- ],
-)
-```
-
-Or alternatively:
-```python
-load("@foo//:requirements.bzl", "all_requirements")
-py_binary(
- name = "baz",
- ...
- deps = [
- ":foo",
- ] + all_requirements,
-)
-```
-""",
- implementation = _pip_repository_impl,
-)
-
-def _impl_whl_library(rctx):
- # pointer to parent repo so these rules rerun if the definitions in requirements.bzl change.
- _parent_repo_label = Label("@{parent}//:requirements.bzl".format(parent = rctx.attr.repo))
- pypath = _construct_pypath(rctx)
- args = [
- rctx.attr.python_interpreter,
- "-m",
- "python.pip_install.parse_requirements_to_bzl.extract_single_wheel",
- "--requirement",
- rctx.attr.requirement,
- "--repo",
- rctx.attr.repo,
- ]
- args = _parse_optional_attrs(rctx, args)
- result = rctx.execute(
- args,
- environment = {
- # Manually construct the PYTHONPATH since we cannot use the toolchain here
- "PYTHONPATH": pypath,
- },
- quiet = rctx.attr.quiet,
- timeout = rctx.attr.timeout,
- )
-
- if result.return_code:
- fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
-
- return
-
-whl_library_attrs = {
- "repo": attr.string(
- mandatory = True,
- doc = "Pointer to parent repo name. Used to make these rules rerun if the parent repo changes.",
- ),
- "requirement": attr.string(
- mandatory = True,
- doc = "Python requirement string describing the package to make available",
- ),
-}
+""
-whl_library_attrs.update(**common_attrs)
+load("//python/private/pypi:group_library.bzl", _group_library = "group_library")
+load("//python/private/pypi:package_annotation.bzl", _package_annotation = "package_annotation")
+load("//python/private/pypi:pip_repository.bzl", _pip_repository = "pip_repository")
+load("//python/private/pypi:whl_library.bzl", _whl_library = "whl_library")
-whl_library = repository_rule(
- attrs = whl_library_attrs,
- doc = """
-Download and extracts a single wheel based into a bazel repo based on the requirement string passed in.
-Instantiated from pip_repository and inherits config options from there.""",
- implementation = _impl_whl_library,
-)
+# Re-exports for backwards compatibility
+group_library = _group_library
+pip_repository = _pip_repository
+whl_library = _whl_library
+package_annotation = _package_annotation
diff --git a/python/pip_install/repositories.bzl b/python/pip_install/repositories.bzl
deleted file mode 100644
index 302ff0ef3b..0000000000
--- a/python/pip_install/repositories.bzl
+++ /dev/null
@@ -1,74 +0,0 @@
-""
-
-load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
-load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
-
-_RULE_DEPS = [
- (
- "pypi__click",
- "https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl",
- "dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc",
- ),
- (
- "pypi__pip",
- "https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl",
- "37fd50e056e2aed635dec96594606f0286640489b0db0ce7607f7e51890372d5",
- ),
- (
- "pypi__pip_tools",
- "https://files.pythonhosted.org/packages/6d/16/75d65bdccd48bb59a08e2bf167b01d8532f65604270d0a292f0f16b7b022/pip_tools-5.5.0-py2.py3-none-any.whl",
- "10841c1e56c234d610d0466447685b9ea4ee4a2c274f858c0ef3c33d9bd0d985",
- ),
- (
- "pypi__pkginfo",
- "https://files.pythonhosted.org/packages/4f/3c/535287349af1b117e082f8e77feca52fbe2fdf61ef1e6da6bcc2a72a3a79/pkginfo-1.6.1-py2.py3-none-any.whl",
- "ce14d7296c673dc4c61c759a0b6c14bae34e34eb819c0017bb6ca5b7292c56e9",
- ),
- (
- "pypi__setuptools",
- "https://files.pythonhosted.org/packages/ab/b5/3679d7c98be5b65fa5522671ef437b792d909cf3908ba54fe9eca5d2a766/setuptools-44.1.0-py2.py3-none-any.whl",
- "992728077ca19db6598072414fb83e0a284aca1253aaf2e24bb1e55ee6db1a30",
- ),
- (
- "pypi__wheel",
- "https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl",
- "78b5b185f0e5763c26ca1e324373aadd49182ca90e825f7853f4b2509215dc0e",
- ),
-]
-
-_GENERIC_WHEEL = """\
-package(default_visibility = ["//visibility:public"])
-
-load("@rules_python//python:defs.bzl", "py_library")
-
-py_library(
- name = "lib",
- srcs = glob(["**/*.py"]),
- data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]),
- # This makes this directory a top-level in the python import
- # search path for anything that depends on this.
- imports = ["."],
-)
-"""
-
-# Collate all the repository names so they can be easily consumed
-all_requirements = [name for (name, _, _) in _RULE_DEPS]
-
-def requirement(pkg):
- return "@pypi__" + pkg + "//:lib"
-
-def pip_install_dependencies():
- """
- Fetch dependencies these rules depend on. Workspaces that use the pip_install rule can call this.
-
- (However we call it from pip_install, making it optional for users to do so.)
- """
- for (name, url, sha256) in _RULE_DEPS:
- maybe(
- http_archive,
- name,
- url = url,
- sha256 = sha256,
- type = "zip",
- build_file_content = _GENERIC_WHEEL,
- )
diff --git a/python/pip_install/requirements.bzl b/python/pip_install/requirements.bzl
index 55e8f61523..6ae3f8fef1 100644
--- a/python/pip_install/requirements.bzl
+++ b/python/pip_install/requirements.bzl
@@ -1,94 +1,19 @@
-"Rules to verify and update pip-compile locked requirements.txt"
-
-load("//python:defs.bzl", "py_binary", "py_test")
-load("//python/pip_install:repositories.bzl", "requirement")
-
-def compile_pip_requirements(
- name,
- extra_args = [],
- visibility = ["//visibility:private"],
- requirements_in = None,
- requirements_txt = None,
- **kwargs):
- """
- Macro creating targets for running pip-compile
-
- Produce a filegroup by default, named "[name]" which can be included in the data
- of some other compile_pip_requirements rule that references these requirements
- (e.g. with `-r ../other/requirements.txt`)
-
- Produce two targets for checking pip-compile:
-
- - validate with `bazel test <login> and
+<password>, which are replaced with their equivalent value
+in the netrc file for the same host name. After formatting, the result is set
+as the value for the Authorization field of the HTTP request.
+
+Example attribute and netrc for a http download to an oauth2 enabled API using a bearer token:
+
+
+auth_patterns = {
+ "storage.cloudprovider.com": "Bearer <password>"
+}
+
+
+netrc:
+
++machine storage.cloudprovider.com + password RANDOM-TOKEN ++ +The final HTTP request would have the following header: + +
+Authorization: Bearer RANDOM-TOKEN ++""" + +# AUTH_ATTRS are used within whl_library and pip bzlmod extension. +AUTH_ATTRS = { + "auth_patterns": attr.string_dict( + doc = _AUTH_PATTERN_DOC, + ), + "netrc": attr.string( + doc = "Location of the .netrc file to use for authentication", + ), +} + +def get_auth(ctx, urls, ctx_attr = None): + """Utility for retrieving netrc-based authentication parameters for repository download rules used in python_repository. + + Args: + ctx(repository_ctx or module_ctx): The extension module_ctx or + repository rule's repository_ctx object. + urls: A list of URLs from which assets will be downloaded. + ctx_attr(struct): The attributes to get the netrc from. When ctx is + repository_ctx, then we will attempt to use repository_ctx.attr + if this is not specified, otherwise we will use the specified + field. The module_ctx attributes are located in the tag classes + so it cannot be retrieved from the context. + + Returns: + dict: A map of authentication parameters by URL. + """ + + # module_ctx does not have attributes, as they are stored in tag classes. Whilst + # the correct behaviour should be to pass the `attr` to the + ctx_attr = ctx_attr or getattr(ctx, "attr", None) + ctx_attr = struct( + netrc = getattr(ctx_attr, "netrc", None), + auth_patterns = getattr(ctx_attr, "auth_patterns", ""), + ) + + if ctx_attr.netrc: + netrc = read_netrc(ctx, ctx_attr.netrc) + elif "NETRC" in ctx.os.environ: + # This can be used on newer bazel versions + if hasattr(ctx, "getenv"): + netrc = read_netrc(ctx, ctx.getenv("NETRC")) + else: + netrc = read_netrc(ctx, ctx.os.environ["NETRC"]) + else: + netrc = read_user_netrc(ctx) + + return use_netrc(netrc, urls, ctx_attr.auth_patterns) diff --git a/python/private/builders.bzl b/python/private/builders.bzl new file mode 100644 index 0000000000..54d46c2af2 --- /dev/null +++ b/python/private/builders.bzl @@ -0,0 +1,197 @@ +# Copyright 2024 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Builders to make building complex objects easier.""" + +load("@bazel_skylib//lib:types.bzl", "types") + +def _DepsetBuilder(order = None): + """Create a builder for a depset. + + Args: + order: {type}`str | None` The order to initialize the depset to, if any. + + Returns: + {type}`DepsetBuilder` + """ + + # buildifier: disable=uninitialized + self = struct( + _order = [order], + add = lambda *a, **k: _DepsetBuilder_add(self, *a, **k), + build = lambda *a, **k: _DepsetBuilder_build(self, *a, **k), + direct = [], + get_order = lambda *a, **k: _DepsetBuilder_get_order(self, *a, **k), + set_order = lambda *a, **k: _DepsetBuilder_set_order(self, *a, **k), + transitive = [], + ) + return self + +def _DepsetBuilder_add(self, *values): + """Add value to the depset. + + Args: + self: {type}`DepsetBuilder` implicitly added. + *values: {type}`depset | list | object` Values to add to the depset. + The values can be a depset, the non-depset value to add, or + a list of such values to add. + + Returns: + {type}`DepsetBuilder` + """ + for value in values: + if types.is_list(value): + for sub_value in value: + if types.is_depset(sub_value): + self.transitive.append(sub_value) + else: + self.direct.append(sub_value) + elif types.is_depset(value): + self.transitive.append(value) + else: + self.direct.append(value) + return self + +def _DepsetBuilder_set_order(self, order): + """Sets the order to use. + + Args: + self: {type}`DepsetBuilder` implicitly added. + order: {type}`str` One of the {obj}`depset` `order` values. + + Returns: + {type}`DepsetBuilder` + """ + self._order[0] = order + return self + +def _DepsetBuilder_get_order(self): + """Gets the depset order that will be used. + + Args: + self: {type}`DepsetBuilder` implicitly added. + + Returns: + {type}`str | None` If not previously set, `None` is returned. + """ + return self._order[0] + +def _DepsetBuilder_build(self): + """Creates a {obj}`depset` from the accumulated values. + + Args: + self: {type}`DepsetBuilder` implicitly added. + + Returns: + {type}`depset` + """ + if not self.direct and len(self.transitive) == 1 and self._order[0] == None: + return self.transitive[0] + else: + kwargs = {} + if self._order[0] != None: + kwargs["order"] = self._order[0] + return depset(direct = self.direct, transitive = self.transitive, **kwargs) + +def _RunfilesBuilder(): + """Creates a `RunfilesBuilder`. + + Returns: + {type}`RunfilesBuilder` + """ + + # buildifier: disable=uninitialized + self = struct( + add = lambda *a, **k: _RunfilesBuilder_add(self, *a, **k), + add_targets = lambda *a, **k: _RunfilesBuilder_add_targets(self, *a, **k), + build = lambda *a, **k: _RunfilesBuilder_build(self, *a, **k), + files = _DepsetBuilder(), + root_symlinks = {}, + runfiles = [], + symlinks = {}, + ) + return self + +def _RunfilesBuilder_add(self, *values): + """Adds a value to the runfiles. + + Args: + self: {type}`RunfilesBuilder` implicitly added. + *values: {type}`File | runfiles | list[File] | depset[File] | list[runfiles]` + The values to add. + + Returns: + {type}`RunfilesBuilder` + """ + for value in values: + if types.is_list(value): + for sub_value in value: + _RunfilesBuilder_add_internal(self, sub_value) + else: + _RunfilesBuilder_add_internal(self, value) + return self + +def _RunfilesBuilder_add_targets(self, targets): + """Adds runfiles from targets + + Args: + self: {type}`RunfilesBuilder` implicitly added. + targets: {type}`list[Target]` targets whose default runfiles + to add. + + Returns: + {type}`RunfilesBuilder` + """ + for t in targets: + self.runfiles.append(t[DefaultInfo].default_runfiles) + return self + +def _RunfilesBuilder_add_internal(self, value): + if _is_file(value): + self.files.add(value) + elif types.is_depset(value): + self.files.add(value) + elif _is_runfiles(value): + self.runfiles.append(value) + else: + fail("Unhandled value: type {}: {}".format(type(value), value)) + +def _RunfilesBuilder_build(self, ctx, **kwargs): + """Creates a {obj}`runfiles` from the accumulated values. + + Args: + self: {type}`RunfilesBuilder` implicitly added. + ctx: {type}`ctx` The rule context to use to create the runfiles object. + **kwargs: additional args to pass along to {obj}`ctx.runfiles`. + + Returns: + {type}`runfiles` + """ + return ctx.runfiles( + transitive_files = self.files.build(), + symlinks = self.symlinks, + root_symlinks = self.root_symlinks, + **kwargs + ).merge_all(self.runfiles) + +# Skylib's types module doesn't have is_file, so roll our own +def _is_file(value): + return type(value) == "File" + +def _is_runfiles(value): + return type(value) == "runfiles" + +builders = struct( + DepsetBuilder = _DepsetBuilder, + RunfilesBuilder = _RunfilesBuilder, +) diff --git a/python/private/builders_util.bzl b/python/private/builders_util.bzl new file mode 100644 index 0000000000..139084f79a --- /dev/null +++ b/python/private/builders_util.bzl @@ -0,0 +1,116 @@ +# Copyright 2025 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Utilities for builders.""" + +load("@bazel_skylib//lib:types.bzl", "types") + +def to_label_maybe(value): + """Converts `value` to a `Label`, maybe. + + The "maybe" qualification is because invalid values for `Label()` + are returned as-is (e.g. None, or special values that might be + used with e.g. the `default` attribute arg). + + Args: + value: {type}`str | Label | None | object` the value to turn into a label, + or return as-is. + + Returns: + {type}`Label | input_value` + """ + if value == None: + return None + if is_label(value): + return value + if types.is_string(value): + return Label(value) + return value + +def is_label(obj): + """Tell if an object is a `Label`.""" + return type(obj) == "Label" + +def kwargs_set_default_ignore_none(kwargs, key, default): + """Normalize None/missing to `default`.""" + existing = kwargs.get(key) + if existing == None: + kwargs[key] = default + +def kwargs_set_default_list(kwargs, key): + """Normalizes None/missing to list.""" + existing = kwargs.get(key) + if existing == None: + kwargs[key] = [] + +def kwargs_set_default_dict(kwargs, key): + """Normalizes None/missing to list.""" + existing = kwargs.get(key) + if existing == None: + kwargs[key] = {} + +def kwargs_set_default_doc(kwargs): + """Sets the `doc` arg default.""" + existing = kwargs.get("doc") + if existing == None: + kwargs["doc"] = "" + +def kwargs_set_default_mandatory(kwargs): + """Sets `False` as the `mandatory` arg default.""" + existing = kwargs.get("mandatory") + if existing == None: + kwargs["mandatory"] = False + +def kwargs_getter(kwargs, key): + """Create a function to get `key` from `kwargs`.""" + return lambda: kwargs.get(key) + +def kwargs_setter(kwargs, key): + """Create a function to set `key` in `kwargs`.""" + + def setter(v): + kwargs[key] = v + + return setter + +def kwargs_getter_doc(kwargs): + """Creates a `kwargs_getter` for the `doc` key.""" + return kwargs_getter(kwargs, "doc") + +def kwargs_setter_doc(kwargs): + """Creates a `kwargs_setter` for the `doc` key.""" + return kwargs_setter(kwargs, "doc") + +def kwargs_getter_mandatory(kwargs): + """Creates a `kwargs_getter` for the `mandatory` key.""" + return kwargs_getter(kwargs, "mandatory") + +def kwargs_setter_mandatory(kwargs): + """Creates a `kwargs_setter` for the `mandatory` key.""" + return kwargs_setter(kwargs, "mandatory") + +def list_add_unique(add_to, others): + """Bulk add values to a list if not already present. + + Args: + add_to: {type}`list[T]` the list to add values to. It is modified + in-place. + others: {type}`collection[collection[T]]` collection of collections of + the values to add. + """ + existing = {v: None for v in add_to} + for values in others: + for value in values: + if value not in existing: + add_to.append(value) diff --git a/python/private/bzlmod_enabled.bzl b/python/private/bzlmod_enabled.bzl new file mode 100644 index 0000000000..84839981a0 --- /dev/null +++ b/python/private/bzlmod_enabled.bzl @@ -0,0 +1,18 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Variable to check if bzlmod is enabled""" + +# When bzlmod is enabled, canonical repos names have @@ in them, while under +# workspace builds, there is never a @@ in labels. +BZLMOD_ENABLED = "@@" in str(Label("//:unused")) diff --git a/examples/legacy_pip_import/helloworld/helloworld.py b/python/private/cc_helper.bzl similarity index 58% rename from examples/legacy_pip_import/helloworld/helloworld.py rename to python/private/cc_helper.bzl index b629e80f28..552b42eae8 100644 --- a/examples/legacy_pip_import/helloworld/helloworld.py +++ b/python/private/cc_helper.bzl @@ -1,4 +1,4 @@ -# Copyright 2017 The Bazel Authors. All rights reserved. +# Copyright 2023 The Bazel Authors. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -11,19 +11,13 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +"""PYTHON RULE IMPLEMENTATION ONLY: Do not use outside of the rule implementations and their tests. -from concurrent import futures +Adapter for accessing Bazel's internal cc_helper. +These may change at any time and are closely coupled to the rule implementation. +""" -class HelloWorld(object): - def __init__(self): - self._threadpool = futures.ThreadPoolExecutor(max_workers=5) +load(":py_internal.bzl", "py_internal") - def SayHello(self): - print("Hello World") - - def SayHelloAsync(self): - self._threadpool.submit(self.SayHello) - - def Stop(self): - self._threadpool.shutdown(wait = True) +cc_helper = getattr(py_internal, "cc_helper", None) diff --git a/python/private/common.bzl b/python/private/common.bzl new file mode 100644 index 0000000000..e49dbad20c --- /dev/null +++ b/python/private/common.bzl @@ -0,0 +1,531 @@ +# Copyright 2022 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Various things common to rule implementations.""" + +load("@bazel_skylib//lib:paths.bzl", "paths") +load("@rules_cc//cc/common:cc_common.bzl", "cc_common") +load("@rules_cc//cc/common:cc_info.bzl", "CcInfo") +load(":cc_helper.bzl", "cc_helper") +load(":py_cc_link_params_info.bzl", "PyCcLinkParamsInfo") +load(":py_info.bzl", "PyInfo", "PyInfoBuilder") +load(":py_internal.bzl", "py_internal") +load(":reexports.bzl", "BuiltinPyInfo") + +_testing = testing +_platform_common = platform_common +_coverage_common = coverage_common +PackageSpecificationInfo = getattr(py_internal, "PackageSpecificationInfo", None) + +# Extensions without the dot +_PYTHON_SOURCE_EXTENSIONS = ["py"] + +# Extensions that mean a file is relevant to Python +PYTHON_FILE_EXTENSIONS = [ + "dll", # Python C modules, Windows specific + "dylib", # Python C modules, Mac specific + "py", + "pyc", + "pyi", + "so", # Python C modules, usually Linux +] + +def create_binary_semantics_struct( + *, + create_executable, + get_cc_details_for_binary, + get_central_uncachable_version_file, + get_coverage_deps, + get_debugger_deps, + get_extra_common_runfiles_for_binary, + get_extra_providers, + get_extra_write_build_data_env, + get_interpreter_path, + get_imports, + get_native_deps_dso_name, + get_native_deps_user_link_flags, + get_stamp_flag, + maybe_precompile, + should_build_native_deps_dso, + should_create_init_files, + should_include_build_data): + """Helper to ensure a semantics struct has all necessary fields. + + Call this instead of a raw call to `struct(...)`; it'll help ensure all + the necessary functions are being correctly provided. + + Args: + create_executable: Callable; creates a binary's executable output. See + py_executable.bzl#py_executable_base_impl for details. + get_cc_details_for_binary: Callable that returns a `CcDetails` struct; see + `create_cc_detail_struct`. + get_central_uncachable_version_file: Callable that returns an optional + Artifact; this artifact is special: it is never cached and is a copy + of `ctx.version_file`; see py_builtins.copy_without_caching + get_coverage_deps: Callable that returns a list of Targets for making + coverage work; only called if coverage is enabled. + get_debugger_deps: Callable that returns a list of Targets that provide + custom debugger support; only called for target-configuration. + get_extra_common_runfiles_for_binary: Callable that returns a runfiles + object of extra runfiles a binary should include. + get_extra_providers: Callable that returns extra providers; see + py_executable.bzl#_create_providers for details. + get_extra_write_build_data_env: Callable that returns a dict[str, str] + of additional environment variable to pass to build data generation. + get_interpreter_path: Callable that returns an optional string, which is + the path to the Python interpreter to use for running the binary. + get_imports: Callable that returns a list of the target's import + paths (from the `imports` attribute, so just the target's own import + path strings, not from dependencies). + get_native_deps_dso_name: Callable that returns a string, which is the + basename (with extension) of the native deps DSO library. + get_native_deps_user_link_flags: Callable that returns a list of strings, + which are any extra linker flags to pass onto the native deps DSO + linking action. + get_stamp_flag: Callable that returns bool of if the --stamp flag was + enabled or not. + maybe_precompile: Callable that may optional precompile the input `.py` + sources and returns the full set of desired outputs derived from + the source files (e.g., both py and pyc, only one of them, etc). + should_build_native_deps_dso: Callable that returns bool; True if + building a native deps DSO is supported, False if not. + should_create_init_files: Callable that returns bool; True if + `__init__.py` files should be generated, False if not. + should_include_build_data: Callable that returns bool; True if + build data should be generated, False if not. + Returns: + A "BinarySemantics" struct. + """ + return struct( + # keep-sorted + create_executable = create_executable, + get_cc_details_for_binary = get_cc_details_for_binary, + get_central_uncachable_version_file = get_central_uncachable_version_file, + get_coverage_deps = get_coverage_deps, + get_debugger_deps = get_debugger_deps, + get_extra_common_runfiles_for_binary = get_extra_common_runfiles_for_binary, + get_extra_providers = get_extra_providers, + get_extra_write_build_data_env = get_extra_write_build_data_env, + get_imports = get_imports, + get_interpreter_path = get_interpreter_path, + get_native_deps_dso_name = get_native_deps_dso_name, + get_native_deps_user_link_flags = get_native_deps_user_link_flags, + get_stamp_flag = get_stamp_flag, + maybe_precompile = maybe_precompile, + should_build_native_deps_dso = should_build_native_deps_dso, + should_create_init_files = should_create_init_files, + should_include_build_data = should_include_build_data, + ) + +def create_library_semantics_struct( + *, + get_cc_info_for_library, + get_imports, + maybe_precompile): + """Create a `LibrarySemantics` struct. + + Call this instead of a raw call to `struct(...)`; it'll help ensure all + the necessary functions are being correctly provided. + + Args: + get_cc_info_for_library: Callable that returns a CcInfo for the library; + see py_library_impl for arg details. + get_imports: Callable; see create_binary_semantics_struct. + maybe_precompile: Callable; see create_binary_semantics_struct. + Returns: + a `LibrarySemantics` struct. + """ + return struct( + # keep sorted + get_cc_info_for_library = get_cc_info_for_library, + get_imports = get_imports, + maybe_precompile = maybe_precompile, + ) + +def create_cc_details_struct( + *, + cc_info_for_propagating, + cc_info_for_self_link, + cc_info_with_extra_link_time_libraries, + extra_runfiles, + cc_toolchain, + feature_config, + **kwargs): + """Creates a CcDetails struct. + + Args: + cc_info_for_propagating: CcInfo that is propagated out of the target + by returning it within a PyCcLinkParamsProvider object. + cc_info_for_self_link: CcInfo that is used when linking for the + binary (or its native deps DSO) itself. This may include extra + information that isn't propagating (e.g. a custom malloc) + cc_info_with_extra_link_time_libraries: CcInfo of extra link time + libraries that MUST come after `cc_info_for_self_link` (or possibly + always last; not entirely clear) when passed to + `link.linking_contexts`. + extra_runfiles: runfiles of extra files needed at runtime, usually as + part of `cc_info_with_extra_link_time_libraries`; should be added to + runfiles. + cc_toolchain: CcToolchain that should be used when building. + feature_config: struct from cc_configure_features(); see + //python/private:py_executable.bzl%cc_configure_features. + **kwargs: Additional keys/values to set in the returned struct. This is to + facilitate extensions with less patching. Any added fields should + pick names that are unlikely to collide if the CcDetails API has + additional fields added. + + Returns: + A `CcDetails` struct. + """ + return struct( + cc_info_for_propagating = cc_info_for_propagating, + cc_info_for_self_link = cc_info_for_self_link, + cc_info_with_extra_link_time_libraries = cc_info_with_extra_link_time_libraries, + extra_runfiles = extra_runfiles, + cc_toolchain = cc_toolchain, + feature_config = feature_config, + **kwargs + ) + +def create_executable_result_struct(*, extra_files_to_build, output_groups, extra_runfiles = None): + """Creates a `CreateExecutableResult` struct. + + This is the return value type of the semantics create_executable function. + + Args: + extra_files_to_build: depset of File; additional files that should be + included as default outputs. + output_groups: dict[str, depset[File]]; additional output groups that + should be returned. + extra_runfiles: A runfiles object of additional runfiles to include. + + Returns: + A `CreateExecutableResult` struct. + """ + return struct( + extra_files_to_build = extra_files_to_build, + output_groups = output_groups, + extra_runfiles = extra_runfiles, + ) + +def csv(values): + """Convert a list of strings to comma separated value string.""" + return ", ".join(sorted(values)) + +def filter_to_py_srcs(srcs): + """Filters .py files from the given list of files""" + + # TODO(b/203567235): Get the set of recognized extensions from + # elsewhere, as there may be others. e.g. Bazel recognizes .py3 + # as a valid extension. + return [f for f in srcs if f.extension == "py"] + +def collect_cc_info(ctx, extra_deps = []): + """Collect C++ information from dependencies for Bazel. + + Args: + ctx: Rule ctx; must have `deps` attribute. + extra_deps: list of Target to also collect C+ information from. + + Returns: + CcInfo provider of merged information. + """ + deps = ctx.attr.deps + if extra_deps: + deps = list(deps) + deps.extend(extra_deps) + cc_infos = [] + for dep in deps: + if CcInfo in dep: + cc_infos.append(dep[CcInfo]) + + if PyCcLinkParamsInfo in dep: + cc_infos.append(dep[PyCcLinkParamsInfo].cc_info) + + return cc_common.merge_cc_infos(cc_infos = cc_infos) + +def collect_imports(ctx, semantics): + """Collect the direct and transitive `imports` strings. + + Args: + ctx: {type}`ctx` the current target ctx + semantics: semantics object for fetching direct imports. + + Returns: + {type}`depset[str]` of import paths + """ + transitive = [] + for dep in ctx.attr.deps: + if PyInfo in dep: + transitive.append(dep[PyInfo].imports) + if BuiltinPyInfo != None and BuiltinPyInfo in dep: + transitive.append(dep[BuiltinPyInfo].imports) + return depset(direct = semantics.get_imports(ctx), transitive = transitive) + +def get_imports(ctx): + """Gets the imports from a rule's `imports` attribute. + + See create_binary_semantics_struct for details about this function. + + Args: + ctx: Rule ctx. + + Returns: + List of strings. + """ + prefix = "{}/{}".format( + ctx.workspace_name, + py_internal.get_label_repo_runfiles_path(ctx.label), + ) + result = [] + for import_str in ctx.attr.imports: + import_str = ctx.expand_make_variables("imports", import_str, {}) + if import_str.startswith("/"): + continue + + # To prevent "escaping" out of the runfiles tree, we normalize + # the path and ensure it doesn't have up-level references. + import_path = paths.normalize("{}/{}".format(prefix, import_str)) + if import_path.startswith("../") or import_path == "..": + fail("Path '{}' references a path above the execution root".format( + import_str, + )) + result.append(import_path) + return result + +def collect_runfiles(ctx, files = depset()): + """Collects the necessary files from the rule's context. + + This presumes the ctx is for a py_binary, py_test, or py_library rule. + + Args: + ctx: rule ctx + files: depset of extra files to include in the runfiles. + Returns: + runfiles necessary for the ctx's target. + """ + return ctx.runfiles( + transitive_files = files, + # This little arg carries a lot of weight, but because Starlark doesn't + # have a way to identify if a target is just a File, the equivalent + # logic can't be re-implemented in pure-Starlark. + # + # Under the hood, it calls the Java `Runfiles#addRunfiles(ctx, + # DEFAULT_RUNFILES)` method, which is the what the Java implementation + # of the Python rules originally did, and the details of how that method + # works have become relied on in various ways. Specifically, what it + # does is visit the srcs, deps, and data attributes in the following + # ways: + # + # For each target in the "data" attribute... + # If the target is a File, then add that file to the runfiles. + # Otherwise, add the target's **data runfiles** to the runfiles. + # + # Note that, contray to best practice, the default outputs of the + # targets in `data` are *not* added, nor are the default runfiles. + # + # This ends up being important for several reasons, some of which are + # specific to Google-internal features of the rules. + # * For Python executables, we have to use `data_runfiles` to avoid + # conflicts for the build data files. Such files have + # target-specific content, but uses a fixed location, so if a + # binary has another binary in `data`, and both try to specify a + # file for that file path, then a warning is printed and an + # arbitrary one will be used. + # * For rules with _entirely_ different sets of files in data runfiles + # vs default runfiles vs default outputs. For example, + # proto_library: documented behavior of this rule is that putting it + # in the `data` attribute will cause the transitive closure of + # `.proto` source files to be included. This set of sources is only + # in the `data_runfiles` (`default_runfiles` is empty). + # * For rules with a _subset_ of files in data runfiles. For example, + # a certain Google rule used for packaging arbitrary binaries will + # generate multiple versions of a binary (e.g. different archs, + # stripped vs un-stripped, etc) in its default outputs, but only + # one of them in the runfiles; this helps avoid large, unused + # binaries contributing to remote executor input limits. + # + # Unfortunately, the above behavior also results in surprising behavior + # in some cases. For example, simple custom rules that only return their + # files in their default outputs won't have their files included. Such + # cases must either return their files in runfiles, or use `filegroup()` + # which will do so for them. + # + # For each target in "srcs" and "deps"... + # Add the default runfiles of the target to the runfiles. While this + # is desirable behavior, it also ends up letting a `py_library` + # be put in `srcs` and still mostly work. + # TODO(b/224640180): Reject py_library et al rules in srcs. + collect_default = True, + ) + +def create_py_info( + ctx, + *, + original_sources, + required_py_files, + required_pyc_files, + implicit_pyc_files, + implicit_pyc_source_files, + imports, + venv_symlinks = []): + """Create PyInfo provider. + + Args: + ctx: rule ctx. + original_sources: `depset[File]`; the original input sources from `srcs` + required_py_files: `depset[File]`; the direct, `.py` sources for the + target that **must** be included by downstream targets. This should + only be Python source files. It should not include pyc files. + required_pyc_files: `depset[File]`; the direct `.pyc` files this target + produces. + implicit_pyc_files: `depset[File]` pyc files that are only used if pyc + collection is enabled. + implicit_pyc_source_files: `depset[File]` source files for implicit pyc + files that are used when the implicit pyc files are not. + implicit_pyc_files: {type}`depset[File]` Implicitly generated pyc files + that a binary can choose to include. + imports: depset of strings; the import path values to propagate. + venv_symlinks: {type}`list[tuple[str, str]]` tuples of + `(runfiles_path, site_packages_path)` for symlinks to create + in the consuming binary's venv site packages. + + Returns: + A tuple of the PyInfo instance and a depset of the + transitive sources collected from dependencies (the latter is only + necessary for deprecated extra actions support). + """ + py_info = PyInfoBuilder.new() + py_info.venv_symlinks.add(venv_symlinks) + py_info.direct_original_sources.add(original_sources) + py_info.direct_pyc_files.add(required_pyc_files) + py_info.direct_pyi_files.add(ctx.files.pyi_srcs) + py_info.transitive_original_sources.add(original_sources) + py_info.transitive_pyc_files.add(required_pyc_files) + py_info.transitive_pyi_files.add(ctx.files.pyi_srcs) + py_info.transitive_implicit_pyc_files.add(implicit_pyc_files) + py_info.transitive_implicit_pyc_source_files.add(implicit_pyc_source_files) + py_info.imports.add(imports) + py_info.merge_has_py2_only_sources(ctx.attr.srcs_version in ("PY2", "PY2ONLY")) + py_info.merge_has_py3_only_sources(ctx.attr.srcs_version in ("PY3", "PY3ONLY")) + + for target in ctx.attr.deps: + # PyInfo may not be present e.g. cc_library rules. + if PyInfo in target or (BuiltinPyInfo != None and BuiltinPyInfo in target): + py_info.merge(_get_py_info(target)) + else: + # TODO(b/228692666): Remove this once non-PyInfo targets are no + # longer supported in `deps`. + files = target.files.to_list() + for f in files: + if f.extension == "py": + py_info.transitive_sources.add(f) + py_info.merge_uses_shared_libraries(cc_helper.is_valid_shared_library_artifact(f)) + for target in ctx.attr.pyi_deps: + # PyInfo may not be present e.g. cc_library rules. + if PyInfo in target or (BuiltinPyInfo != None and BuiltinPyInfo in target): + py_info.merge(_get_py_info(target)) + + deps_transitive_sources = py_info.transitive_sources.build() + py_info.transitive_sources.add(required_py_files) + + # We only look at data to calculate uses_shared_libraries, if it's already + # true, then we don't need to waste time looping over it. + if not py_info.get_uses_shared_libraries(): + # Similar to the above, except we only calculate uses_shared_libraries + for target in ctx.attr.data: + # TODO(b/234730058): Remove checking for PyInfo in data once depot + # cleaned up. + if PyInfo in target or (BuiltinPyInfo != None and BuiltinPyInfo in target): + info = _get_py_info(target) + py_info.merge_uses_shared_libraries(info.uses_shared_libraries) + else: + files = target.files.to_list() + for f in files: + py_info.merge_uses_shared_libraries(cc_helper.is_valid_shared_library_artifact(f)) + if py_info.get_uses_shared_libraries(): + break + if py_info.get_uses_shared_libraries(): + break + + return py_info.build(), deps_transitive_sources, py_info.build_builtin_py_info() + +def _get_py_info(target): + return target[PyInfo] if PyInfo in target or BuiltinPyInfo == None else target[BuiltinPyInfo] + +def create_instrumented_files_info(ctx): + return _coverage_common.instrumented_files_info( + ctx, + source_attributes = ["srcs"], + dependency_attributes = ["deps", "data"], + extensions = _PYTHON_SOURCE_EXTENSIONS, + ) + +def create_output_group_info(transitive_sources, extra_groups): + return OutputGroupInfo( + compilation_prerequisites_INTERNAL_ = transitive_sources, + compilation_outputs = transitive_sources, + **extra_groups + ) + +def maybe_add_test_execution_info(providers, ctx): + """Adds ExecutionInfo, if necessary for proper test execution. + + Args: + providers: Mutable list of providers; may have ExecutionInfo + provider appended. + ctx: Rule ctx. + """ + + # When built for Apple platforms, require the execution to be on a Mac. + # TODO(b/176993122): Remove when bazel automatically knows to run on darwin. + if target_platform_has_any_constraint(ctx, ctx.attr._apple_constraints): + providers.append(_testing.ExecutionInfo({"requires-darwin": ""})) + +_BOOL_TYPE = type(True) + +def is_bool(v): + return type(v) == _BOOL_TYPE + +def target_platform_has_any_constraint(ctx, constraints): + """Check if target platform has any of a list of constraints. + + Args: + ctx: rule context. + constraints: label_list of constraints. + + Returns: + True if target platform has at least one of the constraints. + """ + for constraint in constraints: + constraint_value = constraint[_platform_common.ConstraintValueInfo] + if ctx.target_platform_has_constraint(constraint_value): + return True + return False + +def runfiles_root_path(ctx, short_path): + """Compute a runfiles-root relative path from `File.short_path` + + Args: + ctx: current target ctx + short_path: str, a main-repo relative path from `File.short_path` + + Returns: + {type}`str`, a runflies-root relative path + """ + + # The ../ comes from short_path is for files in other repos. + if short_path.startswith("../"): + return short_path[3:] + else: + return "{}/{}".format(ctx.workspace_name, short_path) diff --git a/python/private/common/py_binary_rule_bazel.bzl b/python/private/common/py_binary_rule_bazel.bzl new file mode 100644 index 0000000000..7858411963 --- /dev/null +++ b/python/private/common/py_binary_rule_bazel.bzl @@ -0,0 +1,6 @@ +"""Stub file for Bazel docs to link to. + +The Bazel docs link to this file, but the implementation was moved. + +Please see: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/defs.html#py_binary +""" diff --git a/python/private/common/py_library_rule_bazel.bzl b/python/private/common/py_library_rule_bazel.bzl new file mode 100644 index 0000000000..be631c9087 --- /dev/null +++ b/python/private/common/py_library_rule_bazel.bzl @@ -0,0 +1,6 @@ +"""Stub file for Bazel docs to link to. + +The Bazel docs link to this file, but the implementation was moved. + +Please see: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/defs.html#py_library +""" diff --git a/python/private/common/py_runtime_rule.bzl b/python/private/common/py_runtime_rule.bzl new file mode 100644 index 0000000000..cadb48c704 --- /dev/null +++ b/python/private/common/py_runtime_rule.bzl @@ -0,0 +1,6 @@ +"""Stub file for Bazel docs to link to. + +The Bazel docs link to this file, but the implementation was moved. + +Please see: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/defs.html#py_runtime +""" diff --git a/python/private/common/py_test_rule_bazel.bzl b/python/private/common/py_test_rule_bazel.bzl new file mode 100644 index 0000000000..c89e3a65c4 --- /dev/null +++ b/python/private/common/py_test_rule_bazel.bzl @@ -0,0 +1,6 @@ +"""Stub file for Bazel docs to link to. + +The Bazel docs link to this file, but the implementation was moved. + +Please see: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/defs.html#py_test +""" diff --git a/python/private/config_settings.bzl b/python/private/config_settings.bzl new file mode 100644 index 0000000000..aff5d016fb --- /dev/null +++ b/python/private/config_settings.bzl @@ -0,0 +1,277 @@ +# Copyright 2024 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""This module is used to construct the config settings in the BUILD file in this same package. +""" + +load("@bazel_skylib//lib:selects.bzl", "selects") +load("@bazel_skylib//rules:common_settings.bzl", "BuildSettingInfo") +load("//python/private:text_util.bzl", "render") +load(":version.bzl", "version") + +_PYTHON_VERSION_FLAG = Label("//python/config_settings:python_version") +_PYTHON_VERSION_MAJOR_MINOR_FLAG = Label("//python/config_settings:python_version_major_minor") + +_DEBUG_ENV_MESSAGE_TEMPLATE = """\ +The current configuration rules_python config flags is: + {flags} + +If the value is missing, then the default value is being used, see documentation: +{docs_url}/python/config_settings +""" + +# Indicates something needs public visibility so that other generated code can +# access it, but it's not intended for general public usage. +_NOT_ACTUALLY_PUBLIC = ["//visibility:public"] + +def construct_config_settings(*, name, default_version, versions, minor_mapping, documented_flags): # buildifier: disable=function-docstring + """Create a 'python_version' config flag and construct all config settings used in rules_python. + + This mainly includes the targets that are used in the toolchain and pip hub + repositories that only match on the 'python_version' flag values. + + Args: + name: {type}`str` A dummy name value that is no-op for now. + default_version: {type}`str` the default value for the `python_version` flag. + versions: {type}`list[str]` A list of versions to build constraint settings for. + minor_mapping: {type}`dict[str, str]` A mapping from `X.Y` to `X.Y.Z` python versions. + documented_flags: {type}`list[str]` The labels of the documented settings + that affect build configuration. + """ + _ = name # @unused + _python_version_flag( + name = _PYTHON_VERSION_FLAG.name, + build_setting_default = default_version, + visibility = ["//visibility:public"], + ) + + _python_version_major_minor_flag( + name = _PYTHON_VERSION_MAJOR_MINOR_FLAG.name, + build_setting_default = "", + visibility = ["//visibility:public"], + ) + + native.config_setting( + name = "is_python_version_unset", + flag_values = {_PYTHON_VERSION_FLAG: ""}, + visibility = ["//visibility:public"], + ) + + _reverse_minor_mapping = {full: minor for minor, full in minor_mapping.items()} + for version in versions: + minor_version = _reverse_minor_mapping.get(version) + if not minor_version: + native.config_setting( + name = "is_python_{}".format(version), + flag_values = {":python_version": version}, + visibility = ["//visibility:public"], + ) + continue + + # Also need to match the minor version when using + name = "is_python_{}".format(version) + native.config_setting( + name = "_" + name, + flag_values = {":python_version": version}, + visibility = ["//visibility:public"], + ) + + # An alias pointing to an underscore-prefixed config_setting_group + # is used because config_setting_group creates + # `is_{version}_N` targets, which are easily confused with the + # `is_{minor}.{micro}` (dot) targets. + selects.config_setting_group( + name = "_{}_group".format(name), + match_any = [ + ":_is_python_{}".format(version), + ":is_python_{}".format(minor_version), + ], + visibility = ["//visibility:private"], + ) + native.alias( + name = name, + actual = "_{}_group".format(name), + visibility = ["//visibility:public"], + ) + + # This matches the raw flag value, e.g. --//python/config_settings:python_version=3.8 + # It's private because matching the concept of e.g. "3.8" value is done + # using the `is_python_X.Y` config setting group, which is aware of the + # minor versions that could match instead. + for minor in minor_mapping.keys(): + native.config_setting( + name = "is_python_{}".format(minor), + flag_values = {_PYTHON_VERSION_MAJOR_MINOR_FLAG: minor}, + visibility = ["//visibility:public"], + ) + + _current_config( + name = "current_config", + build_setting_default = "", + settings = documented_flags + [_PYTHON_VERSION_FLAG.name], + visibility = ["//visibility:private"], + ) + native.config_setting( + name = "is_not_matching_current_config", + # We use the rule above instead of @platforms//:incompatible so that the + # printing of the current env always happens when the _current_config rule + # is executed. + # + # NOTE: This should in practise only happen if there is a missing compatible + # `whl_library` in the hub repo created by `pip.parse`. + flag_values = {"current_config": "will-never-match"}, + # Only public so that PyPI hub repo can access it + visibility = _NOT_ACTUALLY_PUBLIC, + ) + + libc = Label("//python/config_settings:py_linux_libc") + native.config_setting( + name = "_is_py_linux_libc_glibc", + flag_values = {libc: "glibc"}, + visibility = _NOT_ACTUALLY_PUBLIC, + ) + native.config_setting( + name = "_is_py_linux_libc_musl", + flag_values = {libc: "glibc"}, + visibility = _NOT_ACTUALLY_PUBLIC, + ) + freethreaded = Label("//python/config_settings:py_freethreaded") + native.config_setting( + name = "_is_py_freethreaded_yes", + flag_values = {freethreaded: "yes"}, + visibility = _NOT_ACTUALLY_PUBLIC, + ) + native.config_setting( + name = "_is_py_freethreaded_no", + flag_values = {freethreaded: "no"}, + visibility = _NOT_ACTUALLY_PUBLIC, + ) + +def _python_version_flag_impl(ctx): + value = ctx.build_setting_value + return [ + # BuildSettingInfo is the original provider returned, so continue to + # return it for compatibility + BuildSettingInfo(value = value), + # FeatureFlagInfo is returned so that config_setting respects the value + # as returned by this rule instead of as originally seen on the command + # line. + # It is also for Google compatibility, which expects the FeatureFlagInfo + # provider. + config_common.FeatureFlagInfo(value = value), + ] + +_python_version_flag = rule( + implementation = _python_version_flag_impl, + build_setting = config.string(flag = True), + attrs = {}, +) + +def _python_version_major_minor_flag_impl(ctx): + input = _flag_value(ctx.attr._python_version_flag) + if input: + ver = version.parse(input) + value = "{}.{}".format(ver.release[0], ver.release[1]) + else: + value = "" + + return [config_common.FeatureFlagInfo(value = value)] + +_python_version_major_minor_flag = rule( + implementation = _python_version_major_minor_flag_impl, + build_setting = config.string(flag = False), + attrs = { + "_python_version_flag": attr.label( + default = _PYTHON_VERSION_FLAG, + ), + }, +) + +def _flag_value(s): + if config_common.FeatureFlagInfo in s: + return s[config_common.FeatureFlagInfo].value + else: + return s[BuildSettingInfo].value + +def _print_current_config_impl(ctx): + flags = "\n".join([ + "{}: \"{}\"".format(k, v) + for k, v in sorted({ + str(setting.label): _flag_value(setting) + for setting in ctx.attr.settings + }.items()) + ]) + + msg = ctx.attr._template.format( + docs_url = "https://rules-python.readthedocs.io/en/latest/api/rules_python", + flags = render.indent(flags).lstrip(), + ) + if ctx.build_setting_value and ctx.build_setting_value != "fail": + fail("Only 'fail' and empty build setting values are allowed for {}".format( + str(ctx.label), + )) + elif ctx.build_setting_value: + fail(msg) + else: + print(msg) # buildifier: disable=print + + return [config_common.FeatureFlagInfo(value = "")] + +_current_config = rule( + implementation = _print_current_config_impl, + build_setting = config.string(flag = True), + attrs = { + "settings": attr.label_list(mandatory = True), + "_template": attr.string(default = _DEBUG_ENV_MESSAGE_TEMPLATE), + }, +) + +def is_python_version_at_least(name, **kwargs): + flag_name = "_{}_flag".format(name) + native.config_setting( + name = name, + flag_values = { + flag_name: "yes", + }, + ) + _python_version_at_least( + name = flag_name, + visibility = ["//visibility:private"], + **kwargs + ) + +def _python_version_at_least_impl(ctx): + flag_value = ctx.attr._major_minor[config_common.FeatureFlagInfo].value + + # CI is, somehow, getting an empty string for the current flag value. + # How isn't clear. + if not flag_value: + return [config_common.FeatureFlagInfo(value = "no")] + + current = tuple([ + int(x) + for x in flag_value.split(".") + ]) + at_least = tuple([int(x) for x in ctx.attr.at_least.split(".")]) + + value = "yes" if current >= at_least else "no" + return [config_common.FeatureFlagInfo(value = value)] + +_python_version_at_least = rule( + implementation = _python_version_at_least_impl, + attrs = { + "at_least": attr.string(mandatory = True), + "_major_minor": attr.label(default = _PYTHON_VERSION_MAJOR_MINOR_FLAG), + }, +) diff --git a/python/private/coverage.patch b/python/private/coverage.patch new file mode 100644 index 0000000000..051f7fc543 --- /dev/null +++ b/python/private/coverage.patch @@ -0,0 +1,17 @@ +# Because of how coverage is run, the current directory is the first in +# sys.path. This is a problem for the tests, because they may import a module of +# the same name as a module in the current directory. +# +# NOTE @aignas 2023-06-05: we have to do this before anything from coverage gets +# imported. +diff --git a/coverage/__main__.py b/coverage/__main__.py +index ce2d8db..7d7d0a0 100644 +--- a/coverage/__main__.py ++++ b/coverage/__main__.py +@@ -6,5 +6,6 @@ + from __future__ import annotations + + import sys ++sys.path.append(sys.path.pop(0)) + from coverage.cmdline import main + sys.exit(main()) diff --git a/python/private/coverage_deps.bzl b/python/private/coverage_deps.bzl new file mode 100644 index 0000000000..e80e8ee910 --- /dev/null +++ b/python/private/coverage_deps.bzl @@ -0,0 +1,190 @@ +# Copyright 2023 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Dependencies for coverage.py used by the hermetic toolchain. +""" + +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") +load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe") +load("//python/private:version_label.bzl", "version_label") + +# START: maintained by 'bazel run //tools/private/update_deps:update_coverage_deps
.whl for which
-requirements has the dependencies.
-"""),
- "python_interpreter": attr.string(default = "python", doc = """
-The command to run the Python interpreter used when unpacking the wheel.
-"""),
- "requirements": attr.string(doc = """
-The name of the pip_import repository rule from which to load this
-.whl's dependencies.
-"""),
- "whl": attr.label(
- mandatory = True,
- allow_single_file = True,
- doc = """
-The path to the .whl file. The name is expected to follow [this
-convention](https://www.python.org/dev/peps/pep-0427/#file-name-convention)).
-""",
- ),
- "_script": attr.label(
- executable = True,
- default = Label("//tools:whltool.par"),
- cfg = "host",
- ),
- },
- implementation = _whl_impl,
- doc = """A rule for importing `.whl` dependencies into Bazel.
-
-This rule is currently used to implement `pip_import`. It is not intended to
-work standalone, and the interface may change. See `pip_import` for proper
-usage.
-
-This rule imports a `.whl` file as a `py_library`:
-```python
-whl_library(
- name = "foo",
- whl = ":my-whl-file",
- requirements = "name of pip_import rule",
-)
-```
-
-This rule defines `@foo//:pkg` as a `py_library` target.
-""",
-)
diff --git a/renovate.json b/renovate.json
deleted file mode 100644
index ee8c906b91..0000000000
--- a/renovate.json
+++ /dev/null
@@ -1,5 +0,0 @@
-{
- "extends": [
- "config:base"
- ]
-}
diff --git a/sphinxdocs/BUILD.bazel b/sphinxdocs/BUILD.bazel
new file mode 100644
index 0000000000..9ad1e1eef9
--- /dev/null
+++ b/sphinxdocs/BUILD.bazel
@@ -0,0 +1,66 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
+load("//sphinxdocs/private:sphinx.bzl", "repeated_string_list_flag")
+
+package(
+ default_visibility = ["//:__subpackages__"],
+)
+
+# Additional -D values to add to every Sphinx build.
+# This is usually used to override the version when building
+repeated_string_list_flag(
+ name = "extra_defines",
+ build_setting_default = [],
+)
+
+repeated_string_list_flag(
+ name = "extra_env",
+ build_setting_default = [],
+)
+
+# Whether to add the `-q` arg to Sphinx invocations, which determines if
+# stdout has any output or not (logging INFO messages and progress messages).
+# If true, add `-q`. If false, don't add `-q`. This is mostly useful for
+# debugging invocations or developing extensions.
+bool_flag(
+ name = "quiet",
+ build_setting_default = True,
+)
+
+bzl_library(
+ name = "sphinx_bzl",
+ srcs = ["sphinx.bzl"],
+ deps = ["//sphinxdocs/private:sphinx_bzl"],
+)
+
+bzl_library(
+ name = "sphinx_docs_library_bzl",
+ srcs = ["sphinx_docs_library.bzl"],
+ deps = ["//sphinxdocs/private:sphinx_docs_library_macro_bzl"],
+)
+
+bzl_library(
+ name = "sphinx_stardoc_bzl",
+ srcs = ["sphinx_stardoc.bzl"],
+ deps = ["//sphinxdocs/private:sphinx_stardoc_bzl"],
+)
+
+bzl_library(
+ name = "readthedocs_bzl",
+ srcs = ["readthedocs.bzl"],
+ deps = ["//sphinxdocs/private:readthedocs_bzl"],
+)
diff --git a/sphinxdocs/docs/BUILD.bazel b/sphinxdocs/docs/BUILD.bazel
new file mode 100644
index 0000000000..070e0485d7
--- /dev/null
+++ b/sphinxdocs/docs/BUILD.bazel
@@ -0,0 +1,64 @@
+load("//python/private:bzlmod_enabled.bzl", "BZLMOD_ENABLED") # buildifier: disable=bzl-visibility
+load("//sphinxdocs:sphinx_docs_library.bzl", "sphinx_docs_library")
+load("//sphinxdocs:sphinx_stardoc.bzl", "sphinx_stardocs")
+
+package(default_visibility = ["//:__subpackages__"])
+
+# We only build for Linux and Mac because:
+# 1. The actual doc process only runs on Linux
+# 2. Mac is a common development platform, and is close enough to Linux
+# it's feasible to make work.
+# Making CI happy under Windows is too much of a headache, though, so we don't
+# bother with that.
+_TARGET_COMPATIBLE_WITH = select({
+ "@platforms//os:linux": [],
+ "@platforms//os:macos": [],
+ "//conditions:default": ["@platforms//:incompatible"],
+}) if BZLMOD_ENABLED else ["@platforms//:incompatible"]
+
+sphinx_docs_library(
+ name = "docs_lib",
+ deps = [
+ ":artisian_api_docs",
+ ":bzl_docs",
+ ":py_api_srcs",
+ ":regular_docs",
+ ],
+)
+
+sphinx_docs_library(
+ name = "regular_docs",
+ srcs = glob(
+ ["**/*.md"],
+ exclude = ["api/**"],
+ ),
+ prefix = "sphinxdocs/",
+)
+
+sphinx_docs_library(
+ name = "artisian_api_docs",
+ srcs = glob(
+ ["api/**/*.md"],
+ ),
+ prefix = "api/sphinxdocs/",
+ strip_prefix = "sphinxdocs/docs/api/",
+)
+
+sphinx_stardocs(
+ name = "bzl_docs",
+ srcs = [
+ "//sphinxdocs:readthedocs_bzl",
+ "//sphinxdocs:sphinx_bzl",
+ "//sphinxdocs:sphinx_docs_library_bzl",
+ "//sphinxdocs:sphinx_stardoc_bzl",
+ "//sphinxdocs/private:sphinx_docs_library_bzl",
+ ],
+ prefix = "api/sphinxdocs/",
+ target_compatible_with = _TARGET_COMPATIBLE_WITH,
+)
+
+sphinx_docs_library(
+ name = "py_api_srcs",
+ srcs = ["//sphinxdocs/src/sphinx_bzl"],
+ strip_prefix = "sphinxdocs/src/",
+)
diff --git a/sphinxdocs/docs/api/index.md b/sphinxdocs/docs/api/index.md
new file mode 100644
index 0000000000..3420b9180d
--- /dev/null
+++ b/sphinxdocs/docs/api/index.md
@@ -0,0 +1,8 @@
+# sphinxdocs Bazel APIs
+
+API documentation for sphinxdocs Bazel objects.
+
+```{toctree}
+:glob:
+**
+```
diff --git a/sphinxdocs/docs/api/sphinxdocs/index.md b/sphinxdocs/docs/api/sphinxdocs/index.md
new file mode 100644
index 0000000000..bd4e9b6eec
--- /dev/null
+++ b/sphinxdocs/docs/api/sphinxdocs/index.md
@@ -0,0 +1,29 @@
+:::{bzl:currentfile} //sphinxdocs:BUILD.bazel
+:::
+
+# //sphinxdocs
+
+:::{bzl:flag} extra_defines
+Additional `-D` values to add to every Sphinx build.
+
+This is a list flag. Multiple uses are accumulated.
+
+This is most useful for overriding e.g. the version when performing
+release builds.
+:::
+
+:::{bzl:flag} extra_env
+Additional environment variables to for every Sphinx build.
+
+This is a list flag. Multiple uses are accumulated. Values are `key=value`
+format.
+:::
+
+:::{bzl:flag} quiet
+Whether to add the `-q` arg to Sphinx invocations.
+
+This is a boolean flag.
+
+This is useful for debugging invocations or developing extensions. The Sphinx
+`-q` flag causes sphinx to produce additional output on stdout.
+:::
diff --git a/sphinxdocs/docs/api/sphinxdocs/inventories/index.md b/sphinxdocs/docs/api/sphinxdocs/inventories/index.md
new file mode 100644
index 0000000000..a03645ed44
--- /dev/null
+++ b/sphinxdocs/docs/api/sphinxdocs/inventories/index.md
@@ -0,0 +1,11 @@
+:::{bzl:currentfile} //sphinxdocs/inventories:BUILD.bazel
+:::
+
+# //sphinxdocs/inventories
+
+:::{bzl:target} bazel_inventory
+A Sphinx inventory of Bazel objects.
+
+By including this target in your Sphinx build and enabling intersphinx, cross
+references to builtin Bazel objects can be written.
+:::
diff --git a/sphinxdocs/docs/index.md b/sphinxdocs/docs/index.md
new file mode 100644
index 0000000000..bd6448ced9
--- /dev/null
+++ b/sphinxdocs/docs/index.md
@@ -0,0 +1,21 @@
+# Docgen using Sphinx with Bazel
+
+The `sphinxdocs` project allows using Bazel to run Sphinx to generate
+documentation. It comes with:
+
+* Rules for running Sphinx
+* Rules for generating documentation for Starlark code.
+* A Sphinx plugin for documenting Starlark and Bazel objects.
+* Rules for readthedocs build integration.
+
+While it is primarily oriented towards docgen for Starlark code, the core of it
+is agnostic as to what is being documented.
+
+
+```{toctree}
+:hidden:
+
+starlark-docgen
+sphinx-bzl
+readthedocs
+```
diff --git a/sphinxdocs/docs/readthedocs.md b/sphinxdocs/docs/readthedocs.md
new file mode 100644
index 0000000000..c347d19850
--- /dev/null
+++ b/sphinxdocs/docs/readthedocs.md
@@ -0,0 +1,156 @@
+:::{default-domain} bzl
+:::
+
+# Read the Docs integration
+
+The {obj}`readthedocs_install` rule provides support for making it easy
+to build for, and deploy to, Read the Docs. It does this by having Bazel do
+all the work of building, and then the outputs are copied to where Read the Docs
+expects served content to be placed. By having Bazel do the majority of work,
+you have more certainty that the docs you generate locally will match what
+is created in the Read the Docs build environment.
+
+Setting this up is conceptually simple: make the Read the Docs build call `bazel
+run` with the appropriate args. To do this, it requires gluing a couple things
+together, most of which can be copy/pasted from the examples below.
+
+## `.readthedocs.yaml` config
+
+In order for Read the Docs to call our custom commands, we have to use the
+advanced `build.commands` setting of the config file. This needs to do two key
+things:
+1. Install Bazel
+2. Call `bazel run` with the appropriate args.
+
+In the example below, `npm` is used to install Bazelisk and a helper shell
+script, `readthedocs_build.sh` is used to construct the Bazel invocation.
+
+The key purpose of the shell script it to set the
+`--@rules_python//sphinxdocs:extra_env` and
+`--@rules_python//sphinxdocs:extra_defines` flags. These are used to communicate
+`READTHEDOCS*` environment variables and settings to the Bazel invocation.
+
+## BUILD config
+
+In your build file, the {obj}`readthedocs_install` rule handles building the
+docs and copying the output to the Read the Docs output directory
+(`$READTHEDOCS_OUTPUT` environment variable). As input, it takes a `sphinx_docs`
+target (the generated docs).
+
+## conf.py config
+
+Normally, readthedocs will inject extra content into your `conf.py` file
+to make certain integration available (e.g. the version selection flyout).
+However, because our yaml config uses the advanced `build.commands` feature,
+those config injections are disabled and we have to manually re-enable them.
+
+To do this, we modify `conf.py` to detect `READTHEDOCS=True` in the environment
+and perform some additional logic. See the example code below for the
+modifications.
+
+Depending on your theme, you may have to tweak the conf.py; the example is
+based on using the sphinx_rtd_theme.
+
+## Example
+
+```
+# File: .readthedocs.yaml
+version: 2
+
+build:
+ os: "ubuntu-22.04"
+ tools:
+ nodejs: "19"
+ commands:
+ - env
+ - npm install -g @bazel/bazelisk
+ - bazel version
+ # Put the actual action behind a shell script because it's
+ # easier to modify than the yaml config.
+ - docs/readthedocs_build.sh
+```
+
+```
+# File: docs/BUILD
+
+load("@rules_python//sphinxdocs:readthedocs.bzl.bzl", "readthedocs_install")
+readthedocs_install(
+ name = "readthedocs_install",
+ docs = [":docs"],
+)
+```
+
+```
+# File: docs/readthedocs_build.sh
+
+#!/bin/bash
+
+set -eou pipefail
+
+declare -a extra_env
+while IFS='=' read -r -d '' name value; do
+ if [[ "$name" == READTHEDOCS* ]]; then
+ extra_env+=("--@rules_python//sphinxdocs:extra_env=$name=$value")
+ fi
+done < <(env -0)
+
+# In order to get the build number, we extract it from the host name
+extra_env+=("--@rules_python//sphinxdocs:extra_env=HOSTNAME=$HOSTNAME")
+
+set -x
+bazel run \
+ --stamp \
+ "--@rules_python//sphinxdocs:extra_defines=version=$READTHEDOCS_VERSION" \
+ "${extra_env[@]}" \
+ //docs:readthedocs_install
+```
+
+```
+# File: docs/conf.py
+
+# Adapted from the template code:
+# https://github.com/readthedocs/readthedocs.org/blob/main/readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl
+if os.environ.get("READTHEDOCS") == "True":
+ # Must come first because it can interfere with other extensions, according
+ # to the original conf.py template comments
+ extensions.insert(0, "readthedocs_ext.readthedocs")
+
+ if os.environ.get("READTHEDOCS_VERSION_TYPE") == "external":
+ # Insert after the main extension
+ extensions.insert(1, "readthedocs_ext.external_version_warning")
+ readthedocs_vcs_url = (
+ "http://github.com/bazel-contrib/rules_python/pull/{}".format(
+ os.environ.get("READTHEDOCS_VERSION", "")
+ )
+ )
+ # The build id isn't directly available, but it appears to be encoded
+ # into the host name, so we can parse it from that. The format appears
+ # to be `build-X-project-Y-Z`, where:
+ # * X is an integer build id
+ # * Y is an integer project id
+ # * Z is the project name
+ _build_id = os.environ.get("HOSTNAME", "build-0-project-0-rules-python")
+ _build_id = _build_id.split("-")[1]
+ readthedocs_build_url = (
+ f"https://readthedocs.org/projects/rules-python/builds/{_build_id}"
+ )
+
+html_context = {
+ # This controls whether the flyout menu is shown. It is always false
+ # because:
+ # * For local builds, the flyout menu is empty and doesn't show in the
+ # same place as for RTD builds. No point in showing it locally.
+ # * For RTD builds, the flyout menu is always automatically injected,
+ # so having it be True makes the flyout show up twice.
+ "READTHEDOCS": False,
+ "github_version": os.environ.get("READTHEDOCS_GIT_IDENTIFIER", ""),
+ # For local builds, the github link won't work. Disabling it replaces
+ # it with a "view source" link to view the source Sphinx saw, which
+ # is useful for local development.
+ "display_github": os.environ.get("READTHEDOCS") == "True",
+ "commit": os.environ.get("READTHEDOCS_GIT_COMMIT_HASH", "unknown commit"),
+ # Used by readthedocs_ext.external_version_warning extension
+ # This is the PR number being built
+ "current_version": os.environ.get("READTHEDOCS_VERSION", ""),
+}
+```
diff --git a/sphinxdocs/docs/sphinx-bzl.md b/sphinxdocs/docs/sphinx-bzl.md
new file mode 100644
index 0000000000..8376f60679
--- /dev/null
+++ b/sphinxdocs/docs/sphinx-bzl.md
@@ -0,0 +1,328 @@
+# Bazel plugin for Sphinx
+
+The `sphinx_bzl` Python package is a Sphinx plugin that defines a custom domain
+("bzl") in the Sphinx system. This provides first-class integration with Sphinx
+and allows code comments to provide rich information and allows manually writing
+docs for objects that aren't directly representable in bzl source code. For
+example, the fields of a provider can use `:type:` to indicate the type of a
+field, or manually written docs can use the `{bzl:target}` directive to document
+a well known target.
+
+## Configuring Sphinx
+
+To enable the plugin in Sphinx, depend on
+`@rules_python//sphinxdocs/src/sphinx_bzl` and enable it in `conf.py`:
+
+```
+extensions = [
+ "sphinx_bzl.bzl",
+]
+```
+
+## Brief introduction to Sphinx terminology
+
+To aid understanding how to write docs, lets define a few common terms:
+
+* **Role**: A role is the "bzl:obj" part when writing ``{bzl:obj}`ref` ``.
+ Roles mark inline text as needing special processing. There's generally
+ two types of processing: creating cross references, or role-specific custom
+ rendering. For example `{bzl:obj}` will create a cross references, while
+ `{bzl:default-value}` indicates the default value of an argument.
+* **Directive**: A directive is indicated with `:::` and allows defining an
+ entire object and its parts. For example, to describe a function and its
+ arguments, the `:::{bzl:function}` directive is used.
+* **Directive Option**: A directive option is the "type" part when writing
+ `:type:` within a directive. Directive options are how directives are told
+ the meaning of certain values, such as the type of a provider field. Depending
+ on the object being documented, a directive option may be used instead of
+ special role to indicate semantic values.
+
+Most often, you'll be using roles to refer other objects or indicate special
+values in doc strings. For directives, you're likely to only use them when
+manually writing docs to document flags, targets, or other objects that
+`sphinx_stardoc` generates for you.
+
+## MyST vs RST
+
+By default, Sphinx uses ReStructured Text (RST) syntax for its documents.
+Unfortunately, RST syntax is very different than the popular Markdown syntax. To
+bridge the gap, MyST translates Markdown-style syntax into the RST equivalents.
+This allows easily using Markdown in bzl files.
+
+While MyST isn't required for the core `sphinx_bzl` plugin to work, this
+document uses MyST syntax because `sphinx_stardoc` bzl doc gen rule requires
+MyST.
+
+The main difference in syntax is:
+* MyST directives use `:::{name}` with closing `:::` instead of `.. name::` with
+ indented content.
+* MyST roles use `{role:name}` instead of `:role:name:`
+
+## Type expressions
+
+Several roles or fields accept type expressions. Type expressions use
+Python-style annotation syntax to describe data types. For example `None | list[str]`
+describes a type of "None or a list of strings". Each component of the
+expression is parsed and cross reference to its associated type definition.
+
+## Cross references
+
+In brief, to reference bzl objects, use the `bzl:obj` role and use the
+Bazel label string you would use to refer to the object in Bazel (using `%` to
+denote names within a file). For example, to unambiguously refer to `py_binary`:
+
+```
+{bzl:obj}`@rules_python//python:py_binary.bzl%py_binary`
+```
+
+The above is pretty long, so shorter names are also supported, and `sphinx_bzl`
+will try to find something that matches. Additionally, in `.bzl` code, the
+`bzl:` prefix is set as the default. The above can then be shortened to:
+
+```
+{obj}`py_binary`
+```
+
+The text that is displayed can be customized by putting the reference string in
+chevrons (`<>`):
+
+```
+{obj}`the binary rule