Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Patch pip environment with xcode sdk location. #697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jul 1, 2022

Conversation

hrfuller
Copy link
Contributor

@hrfuller hrfuller commented May 9, 2022

Fixes issues in #646 by setting CPPFLAGS to the correct xcode sdks.

PR Checklist

Please check if your PR fulfills the following requirements:

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature (please, look at the "Scope of the project" section in the README.md file)
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • Other... Please describe:

What is the current behavior?

If you use one of the hermetic standalone interpreters provided by rules_python pip fails to build sdists with native code extensions because the sysconfig clang flags provided by the standalone interpreters match the interpreter build environment which is almost always different than the execution environment. We fix this issue by providing the correct libc system headers location via CFLAGS env var.

What is the new behavior?

Pip installing a wheel which needs native code build will succeed.

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

The Windows image in buildkite doesn't have a c++ toolchain available for pip to use to in order to test changes I had to introduce a platform specific requirement into examples/pip_install/requirements.in. In order to appease the platform specific requirements.txt files resulting from compile_pip_requirements I added support for platform specific lock files to compile_pip_requirements. pip-compile can't generate cross platform requirement files from an arbitrary execution platform, so that is left up to the user to generate in their own environments.

@hrfuller hrfuller requested a review from f0rmiga May 9, 2022 20:56
@mattem
Copy link
Contributor

mattem commented May 10, 2022

There were a few other flags I had to override from sysconfig, I'll dig them out.

@hrfuller hrfuller force-pushed the hfuller/646-fix-so-compiles branch from 92f9be0 to 48ba578 Compare May 11, 2022 23:43
@hrfuller hrfuller marked this pull request as ready for review May 30, 2022 19:53
@hrfuller hrfuller requested a review from mattem May 30, 2022 19:56
@mattem
Copy link
Contributor

mattem commented Jun 10, 2022

I also found that we needed to override the include paths set for gcc:

  res = rctx.execute([
      "gcc",
      "-Wp,-v",
      "-xc++",
      "/dev/null",
      "-fsyntax-only",
  ])

  include_paths = [
      paths.normalize(line.replace("(framework directory)", "").strip())
      for line in res.stderr.split("\n")
      if line.startswith(" /usr") or line.startswith(" /Library")
  ]
  include_flags = " ".join(["-I%s" % p for p in include_paths])

And set them on CPPFLAGS.

--sysroot may also need setting so that it applies to libraries too.

@hrfuller
Copy link
Contributor Author

My understanding is that all the standard library includes can be found under the sysroot.

From clang docs:

If the headers are installed in some other system root, the -isysroot option can be used provide a different system root from which the headers will be based. For example, -isysroot /Developer/SDKs/MacOSX10.4u.sdk will look for mylib.h in Developer/SDKs/MacOSX10.4u.sdk/usr/include/mylib.h.

Is the difference that when compiling cpp files you need to set CPPFLAGS in addition to CFLAGS? If so does the -isysroot not apply to cpp compiles?

Though the documentation is a bit confusing clang supports --sysroot in some fashion though its undocumented. I believe that the gcc docs provide some answer to why this is:

-isysroot dir
This option is like the --sysroot option, but applies only to header files (except for Darwin targets, where it applies to both header files and libraries). See the --sysroot option for more information.

-isysroot also applies to the library include path on darwin apparently. Since this change only applies to the darwin platform I think that -isysroot should be enough.

@hrfuller hrfuller force-pushed the hfuller/646-fix-so-compiles branch from 62084a9 to fb84199 Compare June 19, 2022 01:57
@hrfuller
Copy link
Contributor Author

@mattem I think I resolved the issues you brought up except for the extra include flags, which I don't think are required based on docs I've read. If there aren't any blockers can we land this and iterate on any edge cases after? Alternatively if you have a repro of an extension that doesn't build with the current PR I could work off of that.

@hrfuller hrfuller force-pushed the hfuller/646-fix-so-compiles branch from 82c824d to 635d8f3 Compare June 26, 2022 17:58
@hrfuller
Copy link
Contributor Author

@f0rmiga I resolved your comments. I tried using --sysroot instead of -isysroot but it appears to not include the header files in clang when doing so, so we need to use -isysroot anyway.

@f0rmiga
Copy link
Member

f0rmiga commented Jun 27, 2022

@f0rmiga I resolved your comments. I tried using --sysroot instead of -isysroot but it appears to not include the header files in clang when doing so, so we need to use -isysroot anyway.

Oh, interesting!

Thanks for addressing the comments.

@hrfuller hrfuller merged commit ab6940a into main Jul 1, 2022
keith added a commit to keith/envoy that referenced this pull request Jul 18, 2022
This pulls in this commit
bazel-contrib/rules_python#697 which fixes another
issue with envoyproxy#22217 (although
not everything yet)

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
keith added a commit to envoyproxy/envoy that referenced this pull request Jul 18, 2022
This pulls in this commit
bazel-contrib/rules_python#697 which fixes another
issue with #22217 (although
not everything yet)

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
oschaaf pushed a commit to maistra/envoy that referenced this pull request Oct 26, 2022
This pulls in this commit
bazel-contrib/rules_python#697 which fixes another
issue with envoyproxy/envoy#22217 (although
not everything yet)

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
@f0rmiga f0rmiga deleted the hfuller/646-fix-so-compiles branch March 14, 2023 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.