Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit a94deb8

Browse filesBrowse files
linzhpaignas
linzhp
and
aignas
authored
build(gazelle): embed Python zip file (bazel-contrib#1485)
The runtime dependencies of Gazelle Python extension makes it hard to distribute Gazelle binaries: we have to preserve the runfiles structure and distribute it with Gazelle binaries. Instead, we can build a single Python zip file that comes a built-in interpreter, and embed the zip file into the Go binary in compile time and avoid the runtime dependency. Fixes bazel-contrib#1455 --------- Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
1 parent 87a9cf1 commit a94deb8
Copy full SHA for a94deb8

File tree

11 files changed

+90
-55
lines changed
Filter options

11 files changed

+90
-55
lines changed

‎CHANGELOG.md

Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ A brief description of the categories of changes:
2424
* Make `//python/pip_install:pip_repository_bzl` `bzl_library` target internal
2525
as all of the publicly available symbols (etc. `package_annotation`) are
2626
re-exported via `//python:pip_bzl` `bzl_library`.
27+
* Gazelle Python extension no longer has runtime dependencies. Using `GAZELLE_PYTHON_RUNTIME_DEPS` from `@rules_python_gazelle_plugin//:def.bzl` is no longer necessary.
2728

2829
### Fixed
2930

‎examples/build_file_generation/BUILD.bazel

Copy file name to clipboardExpand all lines: examples/build_file_generation/BUILD.bazel
-2Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ load("@bazel_gazelle//:def.bzl", "gazelle")
66
load("@pip//:requirements.bzl", "all_whl_requirements")
77
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
88
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
9-
load("@rules_python_gazelle_plugin//:def.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")
109
load("@rules_python_gazelle_plugin//manifest:defs.bzl", "gazelle_python_manifest")
1110
load("@rules_python_gazelle_plugin//modules_mapping:def.bzl", "modules_mapping")
1211

@@ -56,7 +55,6 @@ gazelle_python_manifest(
5655
# See https://github.com/bazelbuild/bazel-gazelle/blob/master/extend.rst#example
5756
gazelle(
5857
name = "gazelle",
59-
data = GAZELLE_PYTHON_RUNTIME_DEPS,
6058
gazelle = "@rules_python_gazelle_plugin//python:gazelle_binary",
6159
)
6260

‎examples/bzlmod_build_file_generation/BUILD.bazel

Copy file name to clipboardExpand all lines: examples/bzlmod_build_file_generation/BUILD.bazel
-2Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ load("@bazel_gazelle//:def.bzl", "gazelle")
99
load("@pip//:requirements.bzl", "all_whl_requirements")
1010
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
1111
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
12-
load("@rules_python_gazelle_plugin//:def.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")
1312
load("@rules_python_gazelle_plugin//manifest:defs.bzl", "gazelle_python_manifest")
1413
load("@rules_python_gazelle_plugin//modules_mapping:def.bzl", "modules_mapping")
1514

@@ -70,7 +69,6 @@ gazelle_python_manifest(
7069
# See: https://github.com/bazelbuild/bazel-gazelle#fix-and-update
7170
gazelle(
7271
name = "gazelle",
73-
data = GAZELLE_PYTHON_RUNTIME_DEPS,
7472
gazelle = "@rules_python_gazelle_plugin//python:gazelle_binary",
7573
)
7674

‎gazelle/README.md

Copy file name to clipboardExpand all lines: gazelle/README.md
+3-3Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@ Gazelle may be run by Bazel using the gazelle rule, or it may be installed and r
77

88
This directory contains a plugin for
99
[Gazelle](https://github.com/bazelbuild/bazel-gazelle)
10-
that generates BUILD files content for Python code.
10+
that generates BUILD files content for Python code. When Gazelle is run as a command line tool with this plugin, it embeds a Python interpreter resolved during the plugin build.
11+
The behavior of the plugin is slightly different with different version of the interpreter as the Python `stdlib` changes with every minor version release.
12+
Distributors of Gazelle binaries should, therefore, build a Gazelle binary for each OS+CPU architecture+Minor Python version combination they are targeting.
1113

1214
The following instructions are for when you use [bzlmod](https://docs.bazel.build/versions/5.0.0/bzlmod.html).
1315
Please refer to older documentation that includes instructions on how to use Gazelle
@@ -125,7 +127,6 @@ with the rules_python extension included. This typically goes in your root
125127

126128
```starlark
127129
load("@bazel_gazelle//:def.bzl", "gazelle")
128-
load("@rules_python_gazelle_plugin//:def.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")
129130

130131
# Our gazelle target points to the python gazelle binary.
131132
# This is the simple case where we only need one language supported.
@@ -134,7 +135,6 @@ load("@rules_python_gazelle_plugin//:def.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")
134135
# See https://github.com/bazelbuild/bazel-gazelle/blob/master/extend.rst#example
135136
gazelle(
136137
name = "gazelle",
137-
data = GAZELLE_PYTHON_RUNTIME_DEPS,
138138
gazelle = "@rules_python_gazelle_plugin//python:gazelle_binary",
139139
)
140140
```

‎gazelle/def.bzl

Copy file name to clipboardExpand all lines: gazelle/def.bzl
-2Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,4 @@
1616
"""
1717

1818
GAZELLE_PYTHON_RUNTIME_DEPS = [
19-
"@rules_python_gazelle_plugin//python:parse",
20-
"@rules_python_gazelle_plugin//python:std_modules",
2119
]

‎gazelle/python/BUILD.bazel

Copy file name to clipboardExpand all lines: gazelle/python/BUILD.bazel
+14-13Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,7 @@ go_library(
1616
"std_modules.go",
1717
"target.go",
1818
],
19-
data = [
20-
":parse",
21-
":std_modules",
22-
],
19+
embedsrcs = [":helper.zip"],
2320
importpath = "github.com/bazelbuild/rules_python/gazelle/python",
2421
visibility = ["//visibility:public"],
2522
deps = [
@@ -36,33 +33,37 @@ go_library(
3633
"@com_github_emirpasic_gods//lists/singlylinkedlist",
3734
"@com_github_emirpasic_gods//sets/treeset",
3835
"@com_github_emirpasic_gods//utils",
39-
"@io_bazel_rules_go//go/runfiles",
4036
],
4137
)
4238

4339
py_binary(
44-
name = "parse",
45-
srcs = ["parse.py"],
40+
name = "helper",
41+
srcs = [
42+
"__main__.py",
43+
"parse.py",
44+
"std_modules.py",
45+
],
46+
main = "__main__.py",
4647
visibility = ["//visibility:public"],
4748
)
4849

49-
py_binary(
50-
name = "std_modules",
51-
srcs = ["std_modules.py"],
52-
visibility = ["//visibility:public"],
50+
filegroup(
51+
name = "helper.zip",
52+
srcs = [":helper"],
53+
output_group = "python_zip_file",
5354
)
5455

5556
go_test(
5657
name = "python_test",
5758
srcs = ["python_test.go"],
5859
data = [
5960
":gazelle_binary",
60-
":parse",
61-
":std_modules",
61+
":helper",
6262
] + glob(["testdata/**"]),
6363
deps = [
6464
"@bazel_gazelle//testtools:go_default_library",
6565
"@com_github_ghodss_yaml//:yaml",
66+
"@io_bazel_rules_go//go/runfiles:go_default_library",
6667
"@io_bazel_rules_go//go/tools/bazel:go_default_library",
6768
],
6869
)

‎gazelle/python/__main__.py

Copy file name to clipboard
+31Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# parse.py is a long-living program that communicates over STDIN and STDOUT.
16+
# STDIN receives parse requests, one per line. It outputs the parsed modules and
17+
# comments from all the files from each request.
18+
19+
import parse
20+
import std_modules
21+
import sys
22+
23+
if __name__ == "__main__":
24+
if len(sys.argv) < 2:
25+
sys.exit("Please provide subcommand, either print or std_modules")
26+
if sys.argv[1] == "parse":
27+
sys.exit(parse.main(sys.stdin, sys.stdout))
28+
elif sys.argv[1] == "std_modules":
29+
sys.exit(std_modules.main(sys.stdin, sys.stdout))
30+
else:
31+
sys.exit("Unknown subcommand: " + sys.argv[1])

‎gazelle/python/lifecycle.go

Copy file name to clipboardExpand all lines: gazelle/python/lifecycle.go
+26Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,37 @@ package python
1616

1717
import (
1818
"context"
19+
_ "embed"
1920
"github.com/bazelbuild/bazel-gazelle/language"
21+
"log"
22+
"os"
23+
)
24+
25+
var (
26+
//go:embed helper.zip
27+
helperZip []byte
28+
helperPath string
2029
)
2130

2231
type LifeCycleManager struct {
2332
language.BaseLifecycleManager
33+
pyzFilePath string
2434
}
2535

2636
func (l *LifeCycleManager) Before(ctx context.Context) {
37+
helperPath = os.Getenv("GAZELLE_PYTHON_HELPER")
38+
if helperPath == "" {
39+
pyzFile, err := os.CreateTemp("", "python_zip_")
40+
if err != nil {
41+
log.Fatalf("failed to write parser zip: %v", err)
42+
}
43+
defer pyzFile.Close()
44+
helperPath = pyzFile.Name()
45+
l.pyzFilePath = helperPath
46+
if _, err := pyzFile.Write(helperZip); err != nil {
47+
log.Fatalf("cannot write %q: %v", helperPath, err)
48+
}
49+
}
2750
startParserProcess(ctx)
2851
startStdModuleProcess(ctx)
2952
}
@@ -34,4 +57,7 @@ func (l *LifeCycleManager) DoneGeneratingRules() {
3457

3558
func (l *LifeCycleManager) AfterResolvingDeps(ctx context.Context) {
3659
shutdownStdModuleProcess()
60+
if l.pyzFilePath != "" {
61+
os.Remove(l.pyzFilePath)
62+
}
3763
}

‎gazelle/python/parser.go

Copy file name to clipboardExpand all lines: gazelle/python/parser.go
+4-16Lines changed: 4 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ package python
1717
import (
1818
"bufio"
1919
"context"
20+
_ "embed"
2021
"encoding/json"
2122
"fmt"
2223
"io"
@@ -26,7 +27,6 @@ import (
2627
"strings"
2728
"sync"
2829

29-
"github.com/bazelbuild/rules_go/go/runfiles"
3030
"github.com/emirpasic/gods/sets/treeset"
3131
godsutils "github.com/emirpasic/gods/utils"
3232
)
@@ -38,21 +38,9 @@ var (
3838
)
3939

4040
func startParserProcess(ctx context.Context) {
41-
rfiles, err := runfiles.New()
42-
if err != nil {
43-
log.Printf("failed to create a runfiles object: %v\n", err)
44-
os.Exit(1)
45-
}
46-
47-
parseScriptRunfile, err := rfiles.Rlocation("rules_python_gazelle_plugin/python/parse")
48-
if err != nil {
49-
log.Printf("failed to initialize parser: %v\n", err)
50-
os.Exit(1)
51-
}
52-
53-
cmd := exec.CommandContext(ctx, parseScriptRunfile)
54-
cmd.Env = append(os.Environ(), rfiles.Env()...)
55-
41+
// due to #691, we need a system interpreter to boostrap, part of which is
42+
// to locate the hermetic interpreter.
43+
cmd := exec.CommandContext(ctx, "python3", helperPath, "parse")
5644
cmd.Stderr = os.Stderr
5745

5846
stdin, err := cmd.StdinPipe()

‎gazelle/python/python_test.go

Copy file name to clipboardExpand all lines: gazelle/python/python_test.go
+6Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ import (
3131
"time"
3232

3333
"github.com/bazelbuild/bazel-gazelle/testtools"
34+
"github.com/bazelbuild/rules_go/go/runfiles"
3435
"github.com/bazelbuild/rules_go/go/tools/bazel"
3536
"github.com/ghodss/yaml"
3637
)
@@ -159,6 +160,11 @@ func testPath(t *testing.T, name string, files []bazel.RunfileEntry) {
159160
cmd.Stdout = &stdout
160161
cmd.Stderr = &stderr
161162
cmd.Dir = workspaceRoot
163+
helperScript, err := runfiles.Rlocation("rules_python_gazelle_plugin/python/helper")
164+
if err != nil {
165+
t.Fatalf("failed to initialize Python heler: %v", err)
166+
}
167+
cmd.Env = append(os.Environ(), "GAZELLE_PYTHON_HELPER="+helperScript)
162168
if err := cmd.Run(); err != nil {
163169
var e *exec.ExitError
164170
if !errors.As(err, &e) {

‎gazelle/python/std_modules.go

Copy file name to clipboardExpand all lines: gazelle/python/std_modules.go
+5-17Lines changed: 5 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ package python
1717
import (
1818
"bufio"
1919
"context"
20+
_ "embed"
2021
"fmt"
2122
"io"
2223
"log"
@@ -25,8 +26,6 @@ import (
2526
"strconv"
2627
"strings"
2728
"sync"
28-
29-
"github.com/bazelbuild/rules_go/go/runfiles"
3029
)
3130

3231
var (
@@ -39,23 +38,12 @@ var (
3938
func startStdModuleProcess(ctx context.Context) {
4039
stdModulesSeen = make(map[string]struct{})
4140

42-
rfiles, err := runfiles.New()
43-
if err != nil {
44-
log.Printf("failed to create a runfiles object: %v\n", err)
45-
os.Exit(1)
46-
}
47-
48-
stdModulesScriptRunfile, err := rfiles.Rlocation("rules_python_gazelle_plugin/python/std_modules")
49-
if err != nil {
50-
log.Printf("failed to initialize std_modules: %v\n", err)
51-
os.Exit(1)
52-
}
53-
54-
cmd := exec.CommandContext(ctx, stdModulesScriptRunfile)
55-
41+
// due to #691, we need a system interpreter to boostrap, part of which is
42+
// to locate the hermetic interpreter.
43+
cmd := exec.CommandContext(ctx, "python3", helperPath, "std_modules")
5644
cmd.Stderr = os.Stderr
5745
// All userland site-packages should be ignored.
58-
cmd.Env = append([]string{"PYTHONNOUSERSITE=1"}, rfiles.Env()...)
46+
cmd.Env = []string{"PYTHONNOUSERSITE=1"}
5947

6048
stdin, err := cmd.StdinPipe()
6149
if err != nil {

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.