Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 3eb10b2

Browse filesBrowse files
handlerbotmerwokbgehmanarhadthedev
authored and
warsaw
committed
pythongh-66897: Upgrade HTTP CONNECT to protocol HTTP/1.1 (python#8305)
* bpo-22708: Upgrade HTTP CONNECT to protocol HTTP/1.1 (GH-NNNN) Use protocol HTTP/1.1 when sending HTTP CONNECT tunnelling requests; generate Host: headers if one is not already provided (required by HTTP/1.1), convert IDN domains to punycode in HTTP CONNECT requests. * Refactor tests to pass under -bb (fix ByteWarnings); missed some lines >80. * Use consistent 'tunnelling' spelling in Lib/http/client.py * Lib/test/test_httplib: Remove remnant of obsoleted test. * Use dict.copy() not copy.copy() * fix version changed * Update Lib/http/client.py Co-authored-by: bgehman <bgehman@users.noreply.github.com> * Switch to for/else: syntax, as suggested * Don't use for: else: * Sure, fine, w/e * Oops * 1nm to the left --------- Co-authored-by: Éric <merwok@netwok.org> Co-authored-by: bgehman <bgehman@users.noreply.github.com> Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
1 parent 9ca78d7 commit 3eb10b2
Copy full SHA for 3eb10b2

File tree

5 files changed

+167
-21
lines changed
Filter options

5 files changed

+167
-21
lines changed

‎Doc/library/http.client.rst

Copy file name to clipboardExpand all lines: Doc/library/http.client.rst
+12Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -353,6 +353,13 @@ HTTPConnection Objects
353353
The *headers* argument should be a mapping of extra HTTP headers to send with
354354
the CONNECT request.
355355

356+
As HTTP/1.1 is used for HTTP CONNECT tunnelling request, `as per the RFC
357+
<https://tools.ietf.org/html/rfc7231#section-4.3.6>`_, a HTTP ``Host:``
358+
header must be provided, matching the authority-form of the request target
359+
provided as the destination for the CONNECT request. If a HTTP ``Host:``
360+
header is not provided via the headers argument, one is generated and
361+
transmitted automatically.
362+
356363
For example, to tunnel through a HTTPS proxy server running locally on port
357364
8080, we would pass the address of the proxy to the :class:`HTTPSConnection`
358365
constructor, and the address of the host that we eventually want to reach to
@@ -365,6 +372,11 @@ HTTPConnection Objects
365372

366373
.. versionadded:: 3.2
367374

375+
.. versionchanged:: 3.12
376+
HTTP CONNECT tunnelling requests use protocol HTTP/1.1, upgraded from
377+
protocol HTTP/1.0. ``Host:`` HTTP headers are mandatory for HTTP/1.1, so
378+
one will be automatically generated and transmitted if not provided in
379+
the headers argument.
368380

369381
.. method:: HTTPConnection.connect()
370382

‎Lib/http/client.py

Copy file name to clipboardExpand all lines: Lib/http/client.py
+19-6Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -870,27 +870,39 @@ def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
870870
def set_tunnel(self, host, port=None, headers=None):
871871
"""Set up host and port for HTTP CONNECT tunnelling.
872872
873-
In a connection that uses HTTP CONNECT tunneling, the host passed to the
874-
constructor is used as a proxy server that relays all communication to
875-
the endpoint passed to `set_tunnel`. This done by sending an HTTP
873+
In a connection that uses HTTP CONNECT tunnelling, the host passed to
874+
the constructor is used as a proxy server that relays all communication
875+
to the endpoint passed to `set_tunnel`. This done by sending an HTTP
876876
CONNECT request to the proxy server when the connection is established.
877877
878878
This method must be called before the HTTP connection has been
879879
established.
880880
881881
The headers argument should be a mapping of extra HTTP headers to send
882882
with the CONNECT request.
883+
884+
As HTTP/1.1 is used for HTTP CONNECT tunnelling request, as per the RFC
885+
(https://tools.ietf.org/html/rfc7231#section-4.3.6), a HTTP Host:
886+
header must be provided, matching the authority-form of the request
887+
target provided as the destination for the CONNECT request. If a
888+
HTTP Host: header is not provided via the headers argument, one
889+
is generated and transmitted automatically.
883890
"""
884891

885892
if self.sock:
886893
raise RuntimeError("Can't set up tunnel for established connection")
887894

888895
self._tunnel_host, self._tunnel_port = self._get_hostport(host, port)
889896
if headers:
890-
self._tunnel_headers = headers
897+
self._tunnel_headers = headers.copy()
891898
else:
892899
self._tunnel_headers.clear()
893900

901+
if not any(header.lower() == "host" for header in self._tunnel_headers):
902+
encoded_host = self._tunnel_host.encode("idna").decode("ascii")
903+
self._tunnel_headers["Host"] = "%s:%d" % (
904+
encoded_host, self._tunnel_port)
905+
894906
def _get_hostport(self, host, port):
895907
if port is None:
896908
i = host.rfind(':')
@@ -915,8 +927,9 @@ def set_debuglevel(self, level):
915927
self.debuglevel = level
916928

917929
def _tunnel(self):
918-
connect = b"CONNECT %s:%d HTTP/1.0\r\n" % (
919-
self._tunnel_host.encode("ascii"), self._tunnel_port)
930+
connect = b"CONNECT %s:%d %s\r\n" % (
931+
self._tunnel_host.encode("idna"), self._tunnel_port,
932+
self._http_vsn_str.encode("ascii"))
920933
headers = [connect]
921934
for header, value in self._tunnel_headers.items():
922935
headers.append(f"{header}: {value}\r\n".encode("latin-1"))

‎Lib/test/test_httplib.py

Copy file name to clipboardExpand all lines: Lib/test/test_httplib.py
+132-15Lines changed: 132 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2187,11 +2187,12 @@ def test_getting_header_defaultint(self):
21872187
class TunnelTests(TestCase):
21882188
def setUp(self):
21892189
response_text = (
2190-
'HTTP/1.0 200 OK\r\n\r\n' # Reply to CONNECT
2190+
'HTTP/1.1 200 OK\r\n\r\n' # Reply to CONNECT
21912191
'HTTP/1.1 200 OK\r\n' # Reply to HEAD
21922192
'Content-Length: 42\r\n\r\n'
21932193
)
21942194
self.host = 'proxy.com'
2195+
self.port = client.HTTP_PORT
21952196
self.conn = client.HTTPConnection(self.host)
21962197
self.conn._create_connection = self._create_connection(response_text)
21972198

@@ -2203,15 +2204,45 @@ def create_connection(address, timeout=None, source_address=None):
22032204
return FakeSocket(response_text, host=address[0], port=address[1])
22042205
return create_connection
22052206

2206-
def test_set_tunnel_host_port_headers(self):
2207+
def test_set_tunnel_host_port_headers_add_host_missing(self):
22072208
tunnel_host = 'destination.com'
22082209
tunnel_port = 8888
22092210
tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)'}
2211+
tunnel_headers_after = tunnel_headers.copy()
2212+
tunnel_headers_after['Host'] = '%s:%d' % (tunnel_host, tunnel_port)
22102213
self.conn.set_tunnel(tunnel_host, port=tunnel_port,
22112214
headers=tunnel_headers)
22122215
self.conn.request('HEAD', '/', '')
22132216
self.assertEqual(self.conn.sock.host, self.host)
2214-
self.assertEqual(self.conn.sock.port, client.HTTP_PORT)
2217+
self.assertEqual(self.conn.sock.port, self.port)
2218+
self.assertEqual(self.conn._tunnel_host, tunnel_host)
2219+
self.assertEqual(self.conn._tunnel_port, tunnel_port)
2220+
self.assertEqual(self.conn._tunnel_headers, tunnel_headers_after)
2221+
2222+
def test_set_tunnel_host_port_headers_set_host_identical(self):
2223+
tunnel_host = 'destination.com'
2224+
tunnel_port = 8888
2225+
tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)',
2226+
'Host': '%s:%d' % (tunnel_host, tunnel_port)}
2227+
self.conn.set_tunnel(tunnel_host, port=tunnel_port,
2228+
headers=tunnel_headers)
2229+
self.conn.request('HEAD', '/', '')
2230+
self.assertEqual(self.conn.sock.host, self.host)
2231+
self.assertEqual(self.conn.sock.port, self.port)
2232+
self.assertEqual(self.conn._tunnel_host, tunnel_host)
2233+
self.assertEqual(self.conn._tunnel_port, tunnel_port)
2234+
self.assertEqual(self.conn._tunnel_headers, tunnel_headers)
2235+
2236+
def test_set_tunnel_host_port_headers_set_host_different(self):
2237+
tunnel_host = 'destination.com'
2238+
tunnel_port = 8888
2239+
tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)',
2240+
'Host': '%s:%d' % ('example.com', 4200)}
2241+
self.conn.set_tunnel(tunnel_host, port=tunnel_port,
2242+
headers=tunnel_headers)
2243+
self.conn.request('HEAD', '/', '')
2244+
self.assertEqual(self.conn.sock.host, self.host)
2245+
self.assertEqual(self.conn.sock.port, self.port)
22152246
self.assertEqual(self.conn._tunnel_host, tunnel_host)
22162247
self.assertEqual(self.conn._tunnel_port, tunnel_port)
22172248
self.assertEqual(self.conn._tunnel_headers, tunnel_headers)
@@ -2223,17 +2254,96 @@ def test_disallow_set_tunnel_after_connect(self):
22232254
'destination.com')
22242255

22252256
def test_connect_with_tunnel(self):
2226-
self.conn.set_tunnel('destination.com')
2257+
d = {
2258+
b'host': b'destination.com',
2259+
b'port': client.HTTP_PORT,
2260+
}
2261+
self.conn.set_tunnel(d[b'host'].decode('ascii'))
2262+
self.conn.request('HEAD', '/', '')
2263+
self.assertEqual(self.conn.sock.host, self.host)
2264+
self.assertEqual(self.conn.sock.port, self.port)
2265+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2266+
b'Host: %(host)s:%(port)d\r\n\r\n' % d,
2267+
self.conn.sock.data)
2268+
self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d,
2269+
self.conn.sock.data)
2270+
2271+
def test_connect_with_tunnel_with_default_port(self):
2272+
d = {
2273+
b'host': b'destination.com',
2274+
b'port': client.HTTP_PORT,
2275+
}
2276+
self.conn.set_tunnel(d[b'host'].decode('ascii'), port=d[b'port'])
2277+
self.conn.request('HEAD', '/', '')
2278+
self.assertEqual(self.conn.sock.host, self.host)
2279+
self.assertEqual(self.conn.sock.port, self.port)
2280+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2281+
b'Host: %(host)s:%(port)d\r\n\r\n' % d,
2282+
self.conn.sock.data)
2283+
self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d,
2284+
self.conn.sock.data)
2285+
2286+
def test_connect_with_tunnel_with_nonstandard_port(self):
2287+
d = {
2288+
b'host': b'destination.com',
2289+
b'port': 8888,
2290+
}
2291+
self.conn.set_tunnel(d[b'host'].decode('ascii'), port=d[b'port'])
2292+
self.conn.request('HEAD', '/', '')
2293+
self.assertEqual(self.conn.sock.host, self.host)
2294+
self.assertEqual(self.conn.sock.port, self.port)
2295+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2296+
b'Host: %(host)s:%(port)d\r\n\r\n' % d,
2297+
self.conn.sock.data)
2298+
self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s:%(port)d\r\n' % d,
2299+
self.conn.sock.data)
2300+
2301+
# This request is not RFC-valid, but it's been possible with the library
2302+
# for years, so don't break it unexpectedly... This also tests
2303+
# case-insensitivity when injecting Host: headers if they're missing.
2304+
def test_connect_with_tunnel_with_different_host_header(self):
2305+
d = {
2306+
b'host': b'destination.com',
2307+
b'tunnel_host_header': b'example.com:9876',
2308+
b'port': client.HTTP_PORT,
2309+
}
2310+
self.conn.set_tunnel(
2311+
d[b'host'].decode('ascii'),
2312+
headers={'HOST': d[b'tunnel_host_header'].decode('ascii')})
2313+
self.conn.request('HEAD', '/', '')
2314+
self.assertEqual(self.conn.sock.host, self.host)
2315+
self.assertEqual(self.conn.sock.port, self.port)
2316+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2317+
b'HOST: %(tunnel_host_header)s\r\n\r\n' % d,
2318+
self.conn.sock.data)
2319+
self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d,
2320+
self.conn.sock.data)
2321+
2322+
def test_connect_with_tunnel_different_host(self):
2323+
d = {
2324+
b'host': b'destination.com',
2325+
b'port': client.HTTP_PORT,
2326+
}
2327+
self.conn.set_tunnel(d[b'host'].decode('ascii'))
2328+
self.conn.request('HEAD', '/', '')
2329+
self.assertEqual(self.conn.sock.host, self.host)
2330+
self.assertEqual(self.conn.sock.port, self.port)
2331+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2332+
b'Host: %(host)s:%(port)d\r\n\r\n' % d,
2333+
self.conn.sock.data)
2334+
self.assertIn(b'HEAD / HTTP/1.1\r\nHost: %(host)s\r\n' % d,
2335+
self.conn.sock.data)
2336+
2337+
def test_connect_with_tunnel_idna(self):
2338+
dest = '\u03b4\u03c0\u03b8.gr'
2339+
dest_port = b'%s:%d' % (dest.encode('idna'), client.HTTP_PORT)
2340+
expected = b'CONNECT %s HTTP/1.1\r\nHost: %s\r\n\r\n' % (
2341+
dest_port, dest_port)
2342+
self.conn.set_tunnel(dest)
22272343
self.conn.request('HEAD', '/', '')
22282344
self.assertEqual(self.conn.sock.host, self.host)
22292345
self.assertEqual(self.conn.sock.port, client.HTTP_PORT)
2230-
self.assertIn(b'CONNECT destination.com', self.conn.sock.data)
2231-
# issue22095
2232-
self.assertNotIn(b'Host: destination.com:None', self.conn.sock.data)
2233-
self.assertIn(b'Host: destination.com', self.conn.sock.data)
2234-
2235-
# This test should be removed when CONNECT gets the HTTP/1.1 blessing
2236-
self.assertNotIn(b'Host: proxy.com', self.conn.sock.data)
2346+
self.assertIn(expected, self.conn.sock.data)
22372347

22382348
def test_tunnel_connect_single_send_connection_setup(self):
22392349
"""Regresstion test for https://bugs.python.org/issue43332."""
@@ -2253,12 +2363,19 @@ def test_tunnel_connect_single_send_connection_setup(self):
22532363
msg=f'unexpected proxy data sent {proxy_setup_data_sent!r}')
22542364

22552365
def test_connect_put_request(self):
2256-
self.conn.set_tunnel('destination.com')
2366+
d = {
2367+
b'host': b'destination.com',
2368+
b'port': client.HTTP_PORT,
2369+
}
2370+
self.conn.set_tunnel(d[b'host'].decode('ascii'))
22572371
self.conn.request('PUT', '/', '')
22582372
self.assertEqual(self.conn.sock.host, self.host)
2259-
self.assertEqual(self.conn.sock.port, client.HTTP_PORT)
2260-
self.assertIn(b'CONNECT destination.com', self.conn.sock.data)
2261-
self.assertIn(b'Host: destination.com', self.conn.sock.data)
2373+
self.assertEqual(self.conn.sock.port, self.port)
2374+
self.assertIn(b'CONNECT %(host)s:%(port)d HTTP/1.1\r\n'
2375+
b'Host: %(host)s:%(port)d\r\n\r\n' % d,
2376+
self.conn.sock.data)
2377+
self.assertIn(b'PUT / HTTP/1.1\r\nHost: %(host)s\r\n' % d,
2378+
self.conn.sock.data)
22622379

22632380
def test_tunnel_debuglog(self):
22642381
expected_header = 'X-Dummy: 1'

‎Misc/ACKS

Copy file name to clipboardExpand all lines: Misc/ACKS
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -693,6 +693,7 @@ Anders Hammarquist
693693
Mark Hammond
694694
Harald Hanche-Olsen
695695
Manus Hand
696+
Michael Handler
696697
Andreas Hangauer
697698
Milton L. Hankins
698699
Carl Bordum Hansen

‎Misc/NEWS.d/next/Library/2018-07-16-14-10-29.bpo-22708.592iRR.rst

Copy file name to clipboard
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
http.client CONNECT method tunnel improvements: Use HTTP 1.1 protocol; send
2+
a matching Host: header with CONNECT, if one is not provided; convert IDN
3+
domain names to Punycode. Patch by Michael Handler.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.