Incompatibility with urllib3>=2.6.0

The latest https://github.com/urllib3/urllib3/releases/tag/2.6.0 has changed the ContentDecoder class, such that the existing Gzip/Brotli decoders are incompatible and throw an exception. From the release notes: If you use custom decompressors, please make sure to update them to respect the changed API of urllib3.response.ContentDecoder.

Problematic code:

python-storage/google/cloud/storage/_media/requests/download.py

Lines 699 to 766 in 6ed12b9

    
           class _GzipDecoder(urllib3.response.GzipDecoder): 
        
               """Custom subclass of ``urllib3`` decoder for ``gzip``-ed bytes. 
        
               Allows a checksum function to see the compressed bytes before they are 
        
               decoded. This way the checksum of the compressed value can be computed. 
        
               Args: 
        
                   checksum (object): 
        
                       A checksum which will be updated with compressed bytes. 
        
               """ 
        
               def __init__(self, checksum): 
        
                   super().__init__() 
        
                   self._checksum = checksum 
        
               def decompress(self, data): 
        
                   """Decompress the bytes. 
        
                   Args: 
        
                       data (bytes): The compressed bytes to be decompressed. 
        
                   Returns: 
        
                       bytes: The decompressed bytes from ``data``. 
        
                   """ 
        
                   self._checksum.update(data) 
        
                   return super().decompress(data) 
        
           # urllib3.response.BrotliDecoder might not exist depending on whether brotli is 
        
           # installed. 
        
           if hasattr(urllib3.response, "BrotliDecoder"): 
        
               class _BrotliDecoder: 
        
                   """Handler for ``brotli`` encoded bytes. 
        
                   Allows a checksum function to see the compressed bytes before they are 
        
                   decoded. This way the checksum of the compressed value can be computed. 
        
                   Because BrotliDecoder's decompress method is dynamically created in 
        
                   urllib3, a subclass is not practical. Instead, this class creates a 
        
                   captive urllib3.requests.BrotliDecoder instance and acts as a proxy. 
        
                   Args: 
        
                       checksum (object): 
        
                           A checksum which will be updated with compressed bytes. 
        
                   """ 
        
                   def __init__(self, checksum): 
        
                       self._decoder = urllib3.response.BrotliDecoder() 
        
                       self._checksum = checksum 
        
                   def decompress(self, data): 
        
                       """Decompress the bytes. 
        
                       Args: 
        
                           data (bytes): The compressed bytes to be decompressed. 
        
                       Returns: 
        
                           bytes: The decompressed bytes from ``data``. 
        
                       """ 
        
                       self._checksum.update(data) 
        
                       return self._decoder.decompress(data) 
        
                   def flush(self): 
        
                       return self._decoder.flush() 
        
           else:  # pragma: NO COVER 
        
               _BrotliDecoder = None  # type: ignore # pragma: NO COVER

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incompatibility with `urllib3>=2.6.0` #1652

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	class _GzipDecoder(urllib3.response.GzipDecoder):
	"""Custom subclass of ``urllib3`` decoder for ``gzip``-ed bytes.

	Allows a checksum function to see the compressed bytes before they are
	decoded. This way the checksum of the compressed value can be computed.

	Args:
	checksum (object):
	A checksum which will be updated with compressed bytes.
	"""

	def __init__(self, checksum):
	super().__init__()
	self._checksum = checksum

	def decompress(self, data):
	"""Decompress the bytes.

	Args:
	data (bytes): The compressed bytes to be decompressed.

	Returns:
	bytes: The decompressed bytes from ``data``.
	"""
	self._checksum.update(data)
	return super().decompress(data)


	# urllib3.response.BrotliDecoder might not exist depending on whether brotli is
	# installed.
	if hasattr(urllib3.response, "BrotliDecoder"):

	class _BrotliDecoder:
	"""Handler for ``brotli`` encoded bytes.

	Allows a checksum function to see the compressed bytes before they are
	decoded. This way the checksum of the compressed value can be computed.

	Because BrotliDecoder's decompress method is dynamically created in
	urllib3, a subclass is not practical. Instead, this class creates a
	captive urllib3.requests.BrotliDecoder instance and acts as a proxy.

	Args:
	checksum (object):
	A checksum which will be updated with compressed bytes.
	"""

	def __init__(self, checksum):
	self._decoder = urllib3.response.BrotliDecoder()
	self._checksum = checksum

	def decompress(self, data):
	"""Decompress the bytes.

	Args:
	data (bytes): The compressed bytes to be decompressed.

	Returns:
	bytes: The decompressed bytes from ``data``.
	"""
	self._checksum.update(data)
	return self._decoder.decompress(data)

	def flush(self):
	return self._decoder.flush()

	else: # pragma: NO COVER
	_BrotliDecoder = None # type: ignore # pragma: NO COVER

Search code, repositories, users, issues, pull requests...

Incompatibility with urllib3>=2.6.0 #1652

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Incompatibility with `urllib3>=2.6.0` #1652