Bazel rules to transitively fetch and install Python dependencies from a requirements.txt file.
The rules address most of the top packaging issues in bazelbuild/rules_python. This means the rules support common packages such
as tensorflow and google.cloud natively.
- Transitive dependency resolution: #35, #102
- Minimal runtime dependencies: #184
- Support for spreading purelibs: #71
- Support for namespace packages: #14, #55, #65, #93, #189
- Fetches pip packages only for building Python targets: #96
- Reproducible builds: #154, #176
The rules support Python >= 3.5 (the oldest maintained release).
rules_python_external_version = "{COMMIT_SHA}"
http_archive(
name = "rules_python_external",
sha256 = "", # Fill in with correct sha256 of your COMMIT_SHA version
strip_prefix = "rules_python_external-{version}".format(version = rules_python_external_version),
url = "https://github.com/dillon-giacoppo/rules_python_external/archive/v{version}.zip".format(version = rules_python_external_version),
)
# Install the rule dependencies
load("@rules_python_external//:repositories.bzl", "rules_python_external_dependencies")
rules_python_external_dependencies()
load("@rules_python_external//:defs.bzl", "pip_install")
pip_install(
name = "py_deps",
requirements = "//:requirements.txt",
# (Optional) You can provide a python interpreter (by path):
python_interpreter = "/usr/bin/python3.8",
# (Optional) Alternatively you can provide an in-build python interpreter, that is available as a Bazel target.
# This overrides `python_interpreter`.
# Note: You need to set up the interpreter target beforehand (not shown here). Please see the `example` folder for further details.
#python_interpreter_target = "@python_interpreter//:python_bin",
)load("@py_deps//:requirements.bzl", "requirement")
py_binary(
name = "main",
srcs = ["main.py"],
deps = [
requirement("boto3"),
],
)Note that above you do not need to add transitively required packages to deps = [ ... ]
While rules_python_external does not require a transitively-closed requirements.txt file, it is recommended.
But if you want to just have top-level packages listed, that also will work.
Transitively-closed requirements specs are very tedious to produce and maintain manually. To automate the process we
recommend pip-compile from jazzband/pip-tools.
For example, pip-compile takes a requirements.in like this:
boto3~=1.9.227
botocore~=1.12.247
click~=7.0
pip-compile 'compiles' it so you get a transitively-closed requirements.txt like this, which should be passed to
pip_install below:
boto3==1.9.253
botocore==1.12.253
click==7.0
docutils==0.15.2 # via botocore
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.1 # via botocore
s3transfer==0.2.1 # via boto3
six==1.14.0 # via python-dateutil
urllib3==1.25.8 # via botocore
You can find a demo in the example/ directory.
bazel test //...
Here's a (non-exhaustive) list of companies that use rules_python_external in production. Don't see yours? You can add it in a PR!