Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 7740b22

Browse filesBrowse files
UebelAndreUebelAndre
authored
Added support for annotating rendered pip dependencies (bazel-contrib#589)
1 parent 028efa3 commit 7740b22
Copy full SHA for 7740b22

File tree

29 files changed

+989
-69
lines changed
Filter options

29 files changed

+989
-69
lines changed

‎.bazelrc

Copy file name to clipboardExpand all lines: .bazelrc
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/pip_install,examples/pip_parse,examples/py_import,examples/relative_requirements
7-
query --deleted_packages=examples/build_file_generation,examples/pip_install,examples/pip_parse,examples/py_import,examples/relative_requirements
6+
build --deleted_packages=examples/build_file_generation,examples/pip_install,examples/pip_parse,examples/pip_repository_annotations,examples/py_import,examples/relative_requirements
7+
query --deleted_packages=examples/build_file_generation,examples/pip_install,examples/pip_parse,examples/pip_repository_annotations,examples/py_import,examples/relative_requirements
88

99
test --test_output=errors
1010

‎BUILD

Copy file name to clipboardExpand all lines: BUILD
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ filegroup(
3232
"//python:distribution",
3333
"//python/pip_install:distribution",
3434
"//third_party/github.com/bazelbuild/bazel-skylib/lib:distribution",
35+
"//third_party/github.com/bazelbuild/bazel-skylib/rules:distribution",
36+
"//third_party/github.com/bazelbuild/bazel-skylib/rules/private:distribution",
3537
"//tools:distribution",
3638
],
3739
visibility = ["//examples:__pkg__"],

‎docs/pip.md

Copy file name to clipboardExpand all lines: docs/pip.md
+27Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,33 @@ It also generates two targets for running pip-compile:
3535
| kwargs | other bazel attributes passed to the "_test" rule | none |
3636

3737

38+
<a name="#package_annotation"></a>
39+
40+
## package_annotation
41+
42+
<pre>
43+
package_annotation(<a href="#package_annotation-additive_build_content">additive_build_content</a>, <a href="#package_annotation-copy_files">copy_files</a>, <a href="#package_annotation-copy_executables">copy_executables</a>, <a href="#package_annotation-data">data</a>, <a href="#package_annotation-data_exclude_glob">data_exclude_glob</a>,
44+
<a href="#package_annotation-srcs_exclude_glob">srcs_exclude_glob</a>)
45+
</pre>
46+
47+
Annotations to apply to the BUILD file content from package generated from a `pip_repository` rule.
48+
49+
[cf]: https://github.com/bazelbuild/bazel-skylib/blob/main/docs/copy_file_doc.md
50+
51+
52+
**PARAMETERS**
53+
54+
55+
| Name | Description | Default Value |
56+
| :-------------: | :-------------: | :-------------: |
57+
| additive_build_content | Raw text to add to the generated <code>BUILD</code> file of a package. | <code>None</code> |
58+
| copy_files | A mapping of <code>src</code> and <code>out</code> files for [@bazel_skylib//rules:copy_file.bzl][cf] | <code>{}</code> |
59+
| copy_executables | A mapping of <code>src</code> and <code>out</code> files for [@bazel_skylib//rules:copy_file.bzl][cf]. Targets generated here will also be flagged as executable. | <code>{}</code> |
60+
| data | A list of labels to add as <code>data</code> dependencies to the generated <code>py_library</code> target. | <code>[]</code> |
61+
| data_exclude_glob | A list of exclude glob patterns to add as <code>data</code> to the generated <code>py_library</code> target. | <code>[]</code> |
62+
| srcs_exclude_glob | A list of labels to add as <code>srcs</code> to the generated <code>py_library</code> target. | <code>[]</code> |
63+
64+
3865
<a name="#pip_install"></a>
3966

4067
## pip_install

‎examples/BUILD

Copy file name to clipboardExpand all lines: examples/BUILD
+5Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,11 @@ bazel_integration_test(
2727
timeout = "long",
2828
)
2929

30+
bazel_integration_test(
31+
name = "pip_repository_annotations_example",
32+
timeout = "long",
33+
)
34+
3035
bazel_integration_test(
3136
name = "py_import_example",
3237
timeout = "long",

‎examples/pip_repository_annotations/.bazelrc

Copy file name to clipboard
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# https://docs.bazel.build/versions/main/best-practices.html#using-the-bazelrc-file
2+
try-import %workspace%/user.bazelrc

‎examples/pip_repository_annotations/BUILD

Copy file name to clipboard
+30Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
load("@pip_installed//:requirements.bzl", "requirement")
2+
load("@rules_python//python:defs.bzl", "py_test")
3+
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
4+
5+
exports_files(
6+
glob(["data/**"]),
7+
visibility = ["//visibility:public"],
8+
)
9+
10+
# This rule adds a convenient way to update the requirements file.
11+
compile_pip_requirements(
12+
name = "requirements",
13+
extra_args = ["--allow-unsafe"],
14+
)
15+
16+
py_test(
17+
name = "pip_parse_annotations_test",
18+
srcs = ["pip_repository_annotations_test.py"],
19+
env = {"WHEEL_PKG_DIR": "pip_parsed_wheel"},
20+
main = "pip_repository_annotations_test.py",
21+
deps = ["@pip_parsed_wheel//:pkg"],
22+
)
23+
24+
py_test(
25+
name = "pip_install_annotations_test",
26+
srcs = ["pip_repository_annotations_test.py"],
27+
env = {"WHEEL_PKG_DIR": "pip_installed/pypi__wheel"},
28+
main = "pip_repository_annotations_test.py",
29+
deps = [requirement("wheel")],
30+
)

‎examples/pip_repository_annotations/WORKSPACE

Copy file name to clipboard
+58Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
workspace(name = "pip_repository_annotations_example")
2+
3+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
4+
5+
http_archive(
6+
name = "rules_python",
7+
sha256 = "cd6730ed53a002c56ce4e2f396ba3b3be262fd7cb68339f0377a45e8227fe332",
8+
url = "https://github.com/bazelbuild/rules_python/releases/download/0.5.0/rules_python-0.5.0.tar.gz",
9+
)
10+
11+
http_archive(
12+
name = "bazel_skylib",
13+
sha256 = "c6966ec828da198c5d9adbaa94c05e3a1c7f21bd012a0b29ba8ddbccb2c93b0d",
14+
urls = [
15+
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
16+
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
17+
],
18+
)
19+
20+
load("@rules_python//python:pip.bzl", "package_annotation", "pip_install", "pip_parse")
21+
22+
# Here we can see an example of annotations being applied to an arbitrary
23+
# package. For details on `package_annotation` and it's uses, see the
24+
# docs at @rules_python//docs:pip.md`.
25+
ANNOTATIONS = {
26+
"wheel": package_annotation(
27+
additive_build_content = """\
28+
load("@bazel_skylib//rules:write_file.bzl", "write_file")
29+
write_file(
30+
name = "generated_file",
31+
out = "generated_file.txt",
32+
content = ["Hello world from build content file"],
33+
)
34+
""",
35+
copy_executables = {"@pip_repository_annotations_example//:data/copy_executable.py": "copied_content/executable.py"},
36+
copy_files = {"@pip_repository_annotations_example//:data/copy_file.txt": "copied_content/file.txt"},
37+
data = [":generated_file"],
38+
data_exclude_glob = ["*.dist-info/RECORD"],
39+
),
40+
}
41+
42+
# For a more thorough example of `pip_parse`. See `@rules_python//examples/pip_parse`
43+
pip_parse(
44+
name = "pip_parsed",
45+
annotations = ANNOTATIONS,
46+
requirements_lock = "//:requirements.txt",
47+
)
48+
49+
load("@pip_parsed//:requirements.bzl", "install_deps")
50+
51+
install_deps()
52+
53+
# For a more thorough example of `pip_install`. See `@rules_python//examples/pip_install`
54+
pip_install(
55+
name = "pip_installed",
56+
annotations = ANNOTATIONS,
57+
requirements = "//:requirements.txt",
58+
)

‎examples/pip_repository_annotations/data/copy_executable.py

Copy file name to clipboard
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/usr/bin/env python
2+
3+
if __name__ == "__main__":
4+
print("Hello world from copied executable")

‎examples/pip_repository_annotations/data/copy_file.txt

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Hello world from copied file

‎examples/pip_repository_annotations/pip_repository_annotations_test.py

Copy file name to clipboard
+59Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import subprocess
5+
import unittest
6+
from glob import glob
7+
from pathlib import Path
8+
9+
10+
class PipRepositoryAnnotationsTest(unittest.TestCase):
11+
maxDiff = None
12+
13+
def wheel_pkg_dir(self) -> str:
14+
env = os.environ.get("WHEEL_PKG_DIR")
15+
self.assertIsNotNone(env)
16+
return env
17+
18+
def test_build_content_and_data(self):
19+
generated_file = (
20+
Path.cwd() / "external" / self.wheel_pkg_dir() / "generated_file.txt"
21+
)
22+
self.assertTrue(generated_file.exists())
23+
24+
content = generated_file.read_text().rstrip()
25+
self.assertEqual(content, "Hello world from build content file")
26+
27+
def test_copy_files(self):
28+
copied_file = (
29+
Path.cwd() / "external" / self.wheel_pkg_dir() / "copied_content/file.txt"
30+
)
31+
self.assertTrue(copied_file.exists())
32+
33+
content = copied_file.read_text().rstrip()
34+
self.assertEqual(content, "Hello world from copied file")
35+
36+
def test_copy_executables(self):
37+
executable = (
38+
Path.cwd()
39+
/ "external"
40+
/ self.wheel_pkg_dir()
41+
/ "copied_content/executable.py"
42+
)
43+
self.assertTrue(executable.exists())
44+
45+
proc = subprocess.run(
46+
[executable], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
47+
)
48+
stdout = proc.stdout.decode("utf-8").strip()
49+
self.assertEqual(stdout, "Hello world from copied executable")
50+
51+
def test_data_exclude_glob(self):
52+
files = glob("external/" + self.wheel_pkg_dir() + "/wheel-*.dist-info/*")
53+
basenames = [Path(path).name for path in files]
54+
self.assertIn("WHEEL", basenames)
55+
self.assertNotIn("RECORD", basenames)
56+
57+
58+
if __name__ == "__main__":
59+
unittest.main()

‎examples/pip_repository_annotations/requirements.in

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
wheel

‎examples/pip_repository_annotations/requirements.txt

Copy file name to clipboard
+10Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#
2+
# This file is autogenerated by pip-compile
3+
# To update, run:
4+
#
5+
# bazel run //:requirements.update
6+
#
7+
wheel==0.37.1 \
8+
--hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a \
9+
--hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4
10+
# via -r requirements.in

‎python/pip.bzl

Copy file name to clipboardExpand all lines: python/pip.bzl
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,12 @@
1313
# limitations under the License.
1414
"""Import pip requirements into Bazel."""
1515

16-
load("//python/pip_install:pip_repository.bzl", "pip_repository")
16+
load("//python/pip_install:pip_repository.bzl", "pip_repository", _package_annotation = "package_annotation")
1717
load("//python/pip_install:repositories.bzl", "pip_install_dependencies")
1818
load("//python/pip_install:requirements.bzl", _compile_pip_requirements = "compile_pip_requirements")
1919

2020
compile_pip_requirements = _compile_pip_requirements
21+
package_annotation = _package_annotation
2122

2223
def pip_install(requirements, name = "pip", **kwargs):
2324
"""Accepts a `requirements.txt` file and installs the dependencies listed within.

‎python/pip_install/extract_wheels/__init__.py

Copy file name to clipboardExpand all lines: python/pip_install/extract_wheels/__init__.py
+26-7Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,13 @@
1212
import subprocess
1313
import sys
1414

15-
from python.pip_install.extract_wheels.lib import arguments, bazel, requirements
15+
from python.pip_install.extract_wheels.lib import (
16+
annotation,
17+
arguments,
18+
bazel,
19+
requirements,
20+
wheel,
21+
)
1622

1723

1824
def configure_reproducible_wheels() -> None:
@@ -58,6 +64,11 @@ def main() -> None:
5864
required=True,
5965
help="Path to requirements.txt from where to install dependencies",
6066
)
67+
parser.add_argument(
68+
"--annotations",
69+
type=annotation.annotations_map_from_str_path,
70+
help="A json encoded file containing annotations for rendered packages.",
71+
)
6172
arguments.parse_common_args(parser)
6273
args = parser.parse_args()
6374
deserialized_args = dict(vars(args))
@@ -89,18 +100,26 @@ def main() -> None:
89100

90101
repo_label = "@%s" % args.repo
91102

103+
# Locate all wheels
104+
wheels = [whl for whl in glob.glob("*.whl")]
105+
106+
# Collect all annotations
107+
reqs = {whl: wheel.Wheel(whl).name for whl in wheels}
108+
annotations = args.annotations.collect(reqs.values())
109+
92110
targets = [
93111
'"{}{}"'.format(
94112
repo_label,
95113
bazel.extract_wheel(
96-
whl,
97-
extras,
98-
deserialized_args["pip_data_exclude"],
99-
args.enable_implicit_namespace_pkgs,
100-
args.repo_prefix,
114+
wheel_file=whl,
115+
extras=extras,
116+
pip_data_exclude=deserialized_args["pip_data_exclude"],
117+
enable_implicit_namespace_pkgs=args.enable_implicit_namespace_pkgs,
118+
repo_prefix=args.repo_prefix,
119+
annotation=annotations.get(name),
101120
),
102121
)
103-
for whl in glob.glob("*.whl")
122+
for whl, name in reqs.items()
104123
]
105124

106125
with open("requirements.bzl", "w") as requirement_file:

‎python/pip_install/extract_wheels/lib/BUILD

Copy file name to clipboardExpand all lines: python/pip_install/extract_wheels/lib/BUILD
+39Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
load("@rules_python//python:defs.bzl", "py_library", "py_test")
22
load("//python/pip_install:repositories.bzl", "requirement")
3+
load(":annotations_test_helpers.bzl", "package_annotation", "package_annotations_file")
34

45
py_library(
56
name = "lib",
67
srcs = [
8+
"annotation.py",
79
"arguments.py",
810
"bazel.py",
911
"namespace_pkgs.py",
@@ -21,6 +23,43 @@ py_library(
2123
],
2224
)
2325

26+
package_annotations_file(
27+
name = "mock_annotations",
28+
annotations = {
29+
"pkg_a": package_annotation(),
30+
"pkg_b": package_annotation(
31+
data_exclude_glob = [
32+
"*.foo",
33+
"*.bar",
34+
],
35+
),
36+
"pkg_c": package_annotation(
37+
additive_build_content = """\
38+
cc_library(
39+
name = "my_target",
40+
hdrs = glob(["**/*.h"]),
41+
srcs = glob(["**/*.cc"]),
42+
)
43+
""",
44+
data = [":my_target"],
45+
),
46+
"pkg_d": package_annotation(
47+
srcs_exclude_glob = ["pkg_d/tests/**"],
48+
),
49+
},
50+
tags = ["manual"],
51+
)
52+
53+
py_test(
54+
name = "annotations_test",
55+
size = "small",
56+
srcs = ["annotations_test.py"],
57+
data = [":mock_annotations"],
58+
env = {"MOCK_ANNOTATIONS": "$(rootpath :mock_annotations)"},
59+
tags = ["unit"],
60+
deps = [":lib"],
61+
)
62+
2463
py_test(
2564
name = "bazel_test",
2665
size = "small",

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.