Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Exclude autogenerated BUILD file from wheel library runfiles #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

person142
Copy link
Contributor

@person142 person142 commented Mar 27, 2021

PR Checklist

Please check if your PR fulfills the following requirements:

  • Does not include precompiled binaries, eg. .par files. See CONTRIBUTING.md for info
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature (please, look at the "Scope of the project" section in the README.md file)
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • Other... Please describe:

What is the current behavior?

Bazel appears to inject a BUILD file into the root of an external repository. For the non-incremental pip_install the external repository root ends up looking something like this:

BUILD            WORKSPACE        pypi__attrs      requirements.bzl

Since pip_parse doesn't have the extra layers of directories you end up with the external repository looking something like this:

BUILD                  WORKSPACE              attrs-20.3.0.dist-info
BUILD.bazel            attr

The build file is then picked up in the data glob and injected into the runfiles.

Issue Number: #440

What is the new behavior?

This adds the autogenerated BUILD to the exclude so that it doesn't end up in the runfiles. It still won't fix the issue identified in #427 however, but AFAICT that will require changes to Bazel itself to not autogenerate the BUILD files?

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

None

@person142 person142 force-pushed the pip-parsed-no-build branch from 86ffdba to 6b492b7 Compare April 9, 2021 02:25
@person142
Copy link
Contributor Author

@hrfuller a polite ping (don't have permission to request you as a reviewer).

@@ -29,7 +29,10 @@ def generate_build_file_contents(
there may be no Python sources whatsoever (e.g. packages written in Cython: like `pymssql`).
"""

data_exclude = ["*.whl", "**/*.py", "**/* *", "BUILD.bazel", "WORKSPACE"] + pip_data_exclude
data_exclude = (
["*.whl", "**/*.py", "**/* *", "BUILD", "BUILD.bazel", "WORKSPACE"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should WORKSPACE.Bazel be a part of this list too?

Copy link
Contributor

@hrfuller hrfuller Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.bazel.build/versions/master/external.html#working-with-external-dependencies

Looks like WORKSPACE.bazel is an allowed value. We don't use it in our repository rules though.

@hrfuller
Copy link
Contributor

hrfuller commented Apr 15, 2021

I believe this would cause a regression to #427 in pip_parse. I think the correct way to do this is probably to extract the wheels into a subdirectory instead of at the root of the individual repos.

@person142
Copy link
Contributor Author

person142 commented Apr 17, 2021

I believe this would cause a regression to #427 in pip_parse

Hasn’t the regression already happened? An empty BUILD file is already present when using pip_parse.

@person142
Copy link
Contributor Author

person142 commented Apr 17, 2021

Ah, or maybe “this” -> “the current state of affairs” and not “the changes in this PR”, in which case yes I think so.

@hrfuller
Copy link
Contributor

My concern is that excluding "BUILD" would exclude a valid build directory on case insensitive file systems. There wont be a regression ala #427 because we write build file contents into BUILD.bazel now. Bazel "glob" ignores directories by default so if you tried to install the keyring wheel used as example in $427 the build directory would be excluded.

As an aside if you need an immediate workaround you can use the pip_data_excludes

As I mention above, I think the correct way to fix this issue is to create the wheel package in a sub directory.

@person142
Copy link
Contributor Author

There wont be a regression ala #427 because we write build file contents into BUILD.bazel now

Right, but when using pip_parse there is also an autogenerated BUILD file in the directory too (which is what I was trying to convey in #440). i.e. right now there is both BUILD.bazel and BUILD in the generated repos, which should already cause the regression.

Certainly this PR isn't the right fix, and I'm happy to close it-but I believe the regression is already present. Using a sub-directory would be another work around-but at the end of the day this appears to be a Bazel bug-it seems to be adding an automatic empty BUILD file even when a BUILD.bazel file is already present.

@person142
Copy link
Contributor Author

A concrete example of the regression (rules_python is at master):

$ cat keyring_requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile keyring_requirements.in
#
keyring==22.3.0
    # via -r keyring_requirements.in
$ cat WORKSPACE
local_repository(
    name = "rules_python",
    path = "../rules_python",
)

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "keyring",
    requirements_lock = "//:keyring_requirements.txt",
)

load("@keyring//:requirements.bzl", "install_deps")
install_deps()
$ bazelisk build @keyring_pypi__keyring//...
[snip]
NotADirectoryError: [Errno 20] Not a directory: 'build/lib'
)
INFO: Elapsed time: 2.335s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

@person142
Copy link
Contributor Author

person142 commented Apr 17, 2021

Ah, I was mistaken; it's not a Bazel thing; looks like it's this line that inserts the empty build file:

https://github.com/bazelbuild/rules_python/blob/master/python/pip_install/pip_repository.bzl#L15

I'll open a new PR with the proper fix...

@person142
Copy link
Contributor Author

#457 should be a better fix.

@person142 person142 closed this Apr 17, 2021
@person142 person142 deleted the pip-parsed-no-build branch April 17, 2021 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.