Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 7f4cc92

Browse filesBrowse files
mattmoormattmoor
authored
PIP dependency support (#1)
* Check in an initial version of the pip rules. This is broken off of my prototype repository, and there are a handful of TODOs left to resolve. * Remove some vestigial convenience code. Elaborate on why the TODO to rewrite {pip,whl}.sh as Python is harder than it looks. * Incorporate Code Review Feedback * First cut at skydoc-based documentation. * Move whl.sh into a Python script that has unit testing. * Migrate pip.sh into piptool.py This migrates the logic from pip.sh into piptool.py, which should improve portability by removing the bash dependency. This also has the beginnings of wrapping piptool as a closed redistributable that doesn't rely on a system-installed copy of PIP, but instead uses these rules to pull pip into a PAR bundle. Besides needing to work out the details of releasing and redistributing the PAR, we have two unresolved issues: * When bundled as a PAR (vs. py_binary), piptool seems to pick up the system-installed version of pip. * When bundled as a PAR, piptool sometimes sees cert issues resolving requirements (similar to what we see with httplib2). * Address the cert issue in piptool as a PAR. With this change I am able to build/test my PR on my macbook without pip installed. The only additional change I have locally is to switch from running piptool.py as a simple .py file to downloading/using a PAR built from this change. I believe we still have the problem that the .par picks the host's version of pip instead of our embedded copy, but I haven't reverified that issue still exists (this just does nothing to address that issue). * Fix assorted buildifier issues. * Fix a typo in docs * Incorporate @duggelz code review feedback. * Incorporate review feedback move python tools under a top-level rules_python package simplify version_test * Adopt a canonical naming format for imported whl_library rules. `whl_library` rules generated by `pip_import` are now named as: ``` pypi__{distribution}_{version} ``` Substituting illegal characters (e.g. `-`, `.`) with underscores. * Move the piptool dependency into a .PAR file. Add a script for updating the tools and document this and the docs script. * Fix buildifier issues.
1 parent 9160fc7 commit 7f4cc92
Copy full SHA for 7f4cc92
Expand file treeCollapse file tree

37 files changed

+2118
-12
lines changed

‎.ci/rules_python.json

Copy file name to clipboard
+10Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[
2+
{
3+
"variation": "",
4+
"configurations": [
5+
{"node": "linux-x86_64"},
6+
{"node": "ubuntu_16.04-x86_64"},
7+
{"node": "darwin-x86_64"}
8+
]
9+
}
10+
]

‎.gitignore

Copy file name to clipboard
+39Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Compiled Object files
2+
*.slo
3+
*.lo
4+
*.o
5+
*.obj
6+
7+
# Precompiled Headers
8+
*.gch
9+
*.pch
10+
11+
# Compiled Dynamic libraries
12+
*.so
13+
*.dylib
14+
*.dll
15+
16+
# Fortran module files
17+
*.mod
18+
*.smod
19+
20+
# Compiled Static libraries
21+
*.lai
22+
*.la
23+
*.a
24+
*.lib
25+
26+
# Executables
27+
*.exe
28+
*.out
29+
*.app
30+
31+
# Emacs garbage
32+
*~
33+
34+
# Bazel directories
35+
bazel-*
36+
bazel-bin
37+
bazel-genfiles
38+
bazel-out
39+
bazel-testlogs

‎.travis.yml

Copy file name to clipboard
+27Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
sudo: required
2+
dist: trusty
3+
language:
4+
- java
5+
jdk:
6+
- oraclejdk8 # Building Bazel requires JDK8.
7+
addons:
8+
apt:
9+
sources:
10+
- sourceline: 'deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8'
11+
key_url: 'https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg'
12+
packages:
13+
- bazel
14+
15+
install:
16+
- go get -u github.com/bazelbuild/buildifier/buildifier
17+
18+
script:
19+
# Check that all of our tools and samples build
20+
- bazel clean && bazel build //...
21+
# Check that all of our tests pass
22+
- bazel clean && bazel test //...
23+
24+
# Check for issues with the format of our bazel config files.
25+
- buildifier -mode=check $(find . -name BUILD -type f)
26+
- buildifier -mode=check $(find . -name WORKSPACE -type f)
27+
- buildifier -mode=check $(find . -name '*.bzl' -type f)

‎README.md

Copy file name to clipboardExpand all lines: README.md
+84-12Lines changed: 84 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,17 @@
44

55
## Rules
66

7-
* [py_library](#py_library)
8-
* [py_binary](#py_binary)
7+
* [pip_import](docs/python/pip.md#pip_import)
8+
* [py_library](docs/python/python.md#py_library)
9+
* [py_binary](docs/python/python.md#py_binary)
10+
* [py_test](docs/python/python.md#py_test)
911

1012
## Overview
1113

12-
This is a placeholder repository that provides aliases for the native Bazel
13-
python rules. In the future, this will also become the home for rules that
14-
download `pip` packages, and other non-Core Python functionality.
14+
This repository provides Python rules for Bazel. Currently, support for
15+
rules that are available from Bazel core are simple aliases to that bundled
16+
functionality. On top of that, this repository provides support for installing
17+
dependencies typically managed via `pip`.
1518

1619
## Setup
1720

@@ -23,14 +26,19 @@ git_repository(
2326
remote = "https://github.com/bazelbuild/rules_python.git",
2427
commit = "{HEAD}",
2528
)
29+
30+
# Only needed for PIP support:
31+
load("//python:pip.bzl", "pip_repositories")
32+
33+
pip_repositories()
2634
```
2735

2836
Then in your `BUILD` files load the python rules with:
2937

3038
``` python
3139
load(
3240
"@io_bazel_rules_python//python:python.bzl",
33-
"py_binary", "py_library"
41+
"py_binary", "py_library", "py_test",
3442
)
3543

3644
py_binary(
@@ -39,12 +47,76 @@ py_binary(
3947
)
4048
```
4149

42-
<a name="py_library"></a>
43-
## py_library
50+
## Importing `pip` dependencies
51+
52+
These rules are designed to have developers continue using `requirements.txt`
53+
to express their dependencies in a Python idiomatic manner. These dependencies
54+
are imported into the Bazel dependency graph via a two-phased process in
55+
`WORKSPACE`:
56+
57+
```python
58+
load("@io_bazel_rules_python//python:pip.bzl", "pip_import")
59+
60+
# This rule translates the specified requirements.txt into
61+
# @my_deps//:requirements.bzl, which itself exposes a pip_install method.
62+
pip_import(
63+
name = "my_deps",
64+
requirements = "//path/to:requirements.txt",
65+
)
66+
67+
# Load the pip_install symbol for my_deps, and create the dependencies'
68+
# repositories.
69+
load("@my_deps//:requirements.bzl", "pip_install")
70+
pip_install()
71+
```
72+
73+
## Consuming `pip` dependencies
74+
75+
Once a set of dependencies has been imported via `pip_import` and `pip_install`
76+
we can start consuming them in our `py_{binary,library,test}` rules. In support
77+
of this, the generated `requirements.bzl` also contains a `package` method,
78+
which can be used directly in `deps=[]` to reference an imported `py_library`.
79+
80+
```python
81+
load("@my_deps//:requirements.bzl", "package")
82+
83+
py_library(
84+
name = "mylib",
85+
srcs = ["mylib.py"],
86+
deps = [
87+
":myotherlib",
88+
# This takes the name as specified in requirements.txt
89+
package("importeddep"),
90+
]
91+
)
92+
```
93+
94+
## Canonical `whl_library` naming
4495

45-
See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_library).
96+
It is notable that `whl_library` rules imported via `pip_import` are canonically
97+
named, following the pattern: `pypi__{distribution}_{version}`. Characters in
98+
these components that are illegal in Bazel label names (e.g. `-`, `.`) are
99+
replaced with `_`.
46100

47-
<a name="py_binary"></a>
48-
## py_binary
101+
This canonical naming helps avoid redundant work to import the same library
102+
multiple times. It is expected that this naming will remain stable, so folks
103+
should be able to reliably depend directly on e.g. `@pypi__futures_3_1_1//:pkg`
104+
for dependencies, however, it is recommended that folks stick with the `package`
105+
pattern in case the need arises for us to make changes to this format in the
106+
future.
49107

50-
See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_binary).
108+
## Updating `docs/`
109+
110+
All of the content (except `BUILD`) under `docs/` is generated. To update the
111+
documentation simply run this in the root of the repository:
112+
```shell
113+
./update_docs.sh
114+
```
115+
116+
## Updating `tools/`
117+
118+
All of the content (except `BUILD`) under `tools/` is generated. To update the
119+
documentation simply run this in the root of the repository:
120+
```shell
121+
./update_tools.sh
122+
```

‎WORKSPACE

Copy file name to clipboard
+109Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Copyright 2017 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
workspace(name = "io_bazel_rules_python")
15+
16+
# Skydoc stuff
17+
git_repository(
18+
name = "io_bazel_rules_sass",
19+
remote = "https://github.com/bazelbuild/rules_sass.git",
20+
tag = "0.0.2",
21+
)
22+
23+
load("@io_bazel_rules_sass//sass:sass.bzl", "sass_repositories")
24+
25+
sass_repositories()
26+
27+
git_repository(
28+
name = "io_bazel_skydoc",
29+
remote = "https://github.com/bazelbuild/skydoc.git",
30+
tag = "0.1.3",
31+
)
32+
33+
load("@io_bazel_skydoc//skylark:skylark.bzl", "skydoc_repositories")
34+
35+
skydoc_repositories()
36+
37+
# Requirements for building our piptool.
38+
load("//python:pip.bzl", "pip_import")
39+
40+
pip_import(
41+
name = "piptool_deps",
42+
requirements = "//python:requirements.txt",
43+
)
44+
45+
load(
46+
"@piptool_deps//:requirements.bzl",
47+
_piptool_install = "pip_install",
48+
)
49+
50+
_piptool_install()
51+
52+
git_repository(
53+
name = "subpar",
54+
remote = "https://github.com/google/subpar",
55+
tag = "1.0.0",
56+
)
57+
58+
# Test data for WHL tool testing.
59+
http_file(
60+
name = "grpc_whl",
61+
sha256 = "c232d6d168cb582e5eba8e1c0da8d64b54b041dd5ea194895a2fe76050916561",
62+
# From https://pypi.python.org/pypi/grpcio/1.6.0
63+
url = ("https://pypi.python.org/packages/c6/28/" +
64+
"67651b4eabe616b27472c5518f9b2aa3f63beab8f62100b26f05ac428639/" +
65+
"grpcio-1.6.0-cp27-cp27m-manylinux1_i686.whl"),
66+
)
67+
68+
http_file(
69+
name = "futures_whl",
70+
sha256 = "c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f",
71+
# From https://pypi.python.org/pypi/futures
72+
url = ("https://pypi.python.org/packages/a6/1c/" +
73+
"72a18c8c7502ee1b38a604a5c5243aa8c2a64f4bba4e6631b1b8972235dd/" +
74+
"futures-3.1.1-py2-none-any.whl"),
75+
)
76+
77+
http_file(
78+
name = "mock_whl",
79+
sha256 = "5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1",
80+
# From https://pypi.python.org/pypi/mock
81+
url = ("https://pypi.python.org/packages/e6/35/" +
82+
"f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/" +
83+
"mock-2.0.0-py2.py3-none-any.whl"),
84+
)
85+
86+
# Imports for examples
87+
pip_import(
88+
name = "examples_helloworld",
89+
requirements = "//examples/helloworld:requirements.txt",
90+
)
91+
92+
load(
93+
"@examples_helloworld//:requirements.bzl",
94+
_helloworld_install = "pip_install",
95+
)
96+
97+
_helloworld_install()
98+
99+
pip_import(
100+
name = "examples_version",
101+
requirements = "//examples/version:requirements.txt",
102+
)
103+
104+
load(
105+
"@examples_version//:requirements.bzl",
106+
_version_install = "pip_install",
107+
)
108+
109+
_version_install()

‎docs/BUILD

Copy file name to clipboard
+60Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Copyright 2017 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
package(default_visibility = ["//visibility:public"])
15+
16+
licenses(["notice"]) # Apache 2.0
17+
18+
# To regenerate html docs, run:
19+
# ./update_docs.sh
20+
21+
load("@io_bazel_skydoc//skylark:skylark.bzl", "skylark_doc", "skylark_library")
22+
23+
skylark_library(
24+
name = "whl",
25+
srcs = ["//python:whl.bzl"],
26+
)
27+
28+
skylark_library(
29+
name = "pip",
30+
srcs = ["//python:pip.bzl"],
31+
)
32+
33+
skylark_library(
34+
name = "python",
35+
srcs = ["//python:python.bzl"],
36+
)
37+
38+
skylark_doc(
39+
name = "docs-md",
40+
format = "markdown",
41+
overview = True,
42+
site_root = ".",
43+
deps = [
44+
":pip",
45+
":python",
46+
":whl",
47+
],
48+
)
49+
50+
skylark_doc(
51+
name = "docs-html",
52+
format = "html",
53+
overview = True,
54+
site_root = ".",
55+
deps = [
56+
":pip",
57+
":python",
58+
":whl",
59+
],
60+
)

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.