Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

gh-69152: Add _proxy_response_headers attribute to HTTPConnection #26152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 5, 2023
Merged

gh-69152: Add _proxy_response_headers attribute to HTTPConnection #26152

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0200725
add proxy_response_headers attr
nametkin May 13, 2021
7d8cefc
restore debug messages
nametkin May 13, 2021
367893d
add_test
nametkin May 15, 2021
48d9119
refactoring test
nametkin May 16, 2021
ec7f577
add record to NEWS
nametkin May 16, 2021
6d59c02
Merge branch 'main' into add_proxy_response_headers
nametkin May 5, 2023
232240e
Merge branch 'main' into add_proxy_response_headers
orsenthil May 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add proxy_response_headers attr
  • Loading branch information
@nametkin
nametkin committed Apr 29, 2023
commit 02007252e81ca61b27f6a45b202f1c3c6d19576a
15 changes: 3 additions & 12 deletions 15 Lib/http/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -858,6 +858,7 @@ def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
self._tunnel_host = None
self._tunnel_port = None
self._tunnel_headers = {}
self._proxy_response_headers = None

(self.host, self.port) = self._get_hostport(host, port)

Expand Down Expand Up @@ -943,21 +944,11 @@ def _tunnel(self):
response = self.response_class(self.sock, method=self._method)
(version, code, message) = response._read_status()

self._proxy_response_headers = parse_headers(response.fp)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to how it is done in lines 337-341


if code != http.HTTPStatus.OK:
self.close()
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
while True:
line = response.fp.readline(_MAXLINE + 1)
if len(line) > _MAXLINE:
raise LineTooLong("header line")
if not line:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition if not line: isn't captured in parse_headers call.
Additional check of _MAXHEADERS is verified.
So, this isn't a strict 1:1 replacement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add the if not line condition in _reader_headers method for equivalence.

However, given that _tunnel method already prints headers in line 960 in print('header:', line.decode()), I am trying to understand, what additional benefit this patch brings

The discussion in #69152 seems adding additional states to the _tunnel method, and adding debug response to those states.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orsenthil, thanks for the review!
Regarding condition if not line. It seems to me that this is now a useless part of the code. Because the only variant of the line value in which this condition will be true, if I'm not mistaken, is line = b" (because the method readline can't return None or just an empty string). And for the equality of line to the value of b'', we check a little below.

I looked at the history, these lines were added when there was no equality check for the empty binary object below. That is, at some point they made sense, but after this commit, they ceased to be necessary.

But we can add this check to the _read_headers method as a precaution. Do you think it should be done?

Copy link
Contributor Author

@nametkin nametkin Apr 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. As for the benefits of the changes being made. Firstly, headers are needed in the situations described in the comment Add tunnel CONNECT response headers to httplib / http.client #69152 (comment). And secondly, now headers gets into the debug-logs only if we were able to go beyond this line: https://github.com/python/cpython/blob/main/Lib/http/client.py#L948.
    But if we did not pass the authorization headers we need, then we will get an OSError without information about what happened, we will not be able to find out what authentication data is required. With my PR, I would like to take the first step towards solving the problems described here Add support for digest authentication with an HTTP proxy psf/requests#2526, Allow custom authentication (in particular NTLM) to proxies  psf/requests#1582, http proxy negotiate/gssapi authentication? requests/requests-kerberos#83

Maybe then it would be possible to start returning not OSError, but some custom error that can be catched in the library code (urllib3, requests) and automatically prepare data for authentication based on the information contained in _proxy_response_headers. Similar to how it is done in https://github.com/requests/requests-kerberos/pull/149/files.

Maybe you know ways to solve these problems in a different way, I will be glad of any ideas and suggestions.
How @vadmium's proposal will solve the problem of the lack of any information about what went wrong when creating the tunnel, I did not understand.

# for sites which EOF without sending a trailer
break
if line in (b'\r\n', b'\n', b''):
break

if self.debuglevel > 0:
print('header:', line.decode())

def connect(self):
"""Connect to the host and port specified in __init__."""
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.