Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 741d3fd

Browse filesBrowse files
daniellehanksdaniellehanks
authored
feat: add support for Etag headers on reads (#489)
Support conditional requests based on ETag for read operations (`reload`, `exists`, `download_*`). My own testing seems to indicate that the JSON API does not support ETag If-Match/If-None-Match headers on modify requests (`patch`, `delete`, etc.), please correct me if I am mistaken. This part two of #451. Part one in #488. Fixes #451 🦕
1 parent 49ba14c commit 741d3fd
Copy full SHA for 741d3fd

File tree

Expand file treeCollapse file tree

11 files changed

+701
-16
lines changed
Filter options
Expand file treeCollapse file tree

11 files changed

+701
-16
lines changed

‎docs/generation_metageneration.rst

Copy file name to clipboardExpand all lines: docs/generation_metageneration.rst
+44-4Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,31 @@
1-
Conditional Requests Via Generation / Metageneration Preconditions
2-
==================================================================
1+
Conditional Requests Via ETag / Generation / Metageneration Preconditions
2+
=========================================================================
33

44
Preconditions tell Cloud Storage to only perform a request if the
5-
:ref:`generation <concept-generation>` or
5+
:ref:`ETag <concept-etag>`, :ref:`generation <concept-generation>`, or
66
:ref:`metageneration <concept-metageneration>` number of the affected object
7-
meets your precondition criteria. These checks of the generation and
7+
meets your precondition criteria. These checks of the ETag, generation, and
88
metageneration numbers ensure that the object is in the expected state,
99
allowing you to perform safe read-modify-write updates and conditional
1010
operations on objects
1111

1212
Concepts
1313
--------
1414

15+
.. _concept-etag:
16+
17+
ETag
18+
::::::::::::::
19+
20+
An ETag is returned as part of the response header whenever a resource is
21+
returned, as well as included in the resource itself. Users should make no
22+
assumptions about the value used in an ETag except that it changes whenever the
23+
underlying data changes, per the
24+
`specification <https://tools.ietf.org/html/rfc7232#section-2.3>`_
25+
26+
The ``ETag`` attribute is set by the GCS back-end, and is read-only in the
27+
client library.
28+
1529
.. _concept-metageneration:
1630

1731
Metageneration
@@ -59,6 +73,32 @@ See also
5973
Conditional Parameters
6074
----------------------
6175

76+
.. _using-if-etag-match:
77+
78+
Using ``if_etag_match``
79+
:::::::::::::::::::::::::::::
80+
81+
Passing the ``if_etag_match`` parameter to a method which retrieves a
82+
blob resource (e.g.,
83+
:meth:`Blob.reload <google.cloud.storage.blob.Blob.reload>`)
84+
makes the operation conditional on whether the blob's current ``ETag`` matches
85+
the given value. This parameter is not supported for modification (e.g.,
86+
:meth:`Blob.update <google.cloud.storage.blob.Blob.update>`).
87+
88+
89+
.. _using-if-etag-not-match:
90+
91+
Using ``if_etag_not_match``
92+
:::::::::::::::::::::::::::::
93+
94+
Passing the ``if_etag_not_match`` parameter to a method which retrieves a
95+
blob resource (e.g.,
96+
:meth:`Blob.reload <google.cloud.storage.blob.Blob.reload>`)
97+
makes the operation conditional on whether the blob's current ``ETag`` matches
98+
the given value. This parameter is not supported for modification (e.g.,
99+
:meth:`Blob.update <google.cloud.storage.blob.Blob.update>`).
100+
101+
62102
.. _using-if-generation-match:
63103

64104
Using ``if_generation_match``

‎google/cloud/storage/_helpers.py

Copy file name to clipboardExpand all lines: google/cloud/storage/_helpers.py
+39-3Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
from datetime import datetime
2323
import os
2424

25+
from six import string_types
2526
from six.moves.urllib.parse import urlsplit
2627
from google import resumable_media
2728
from google.cloud.storage.constants import _DEFAULT_TIMEOUT
@@ -34,6 +35,12 @@
3435

3536
_DEFAULT_STORAGE_HOST = u"https://storage.googleapis.com"
3637

38+
# etag match parameters in snake case and equivalent header
39+
_ETAG_MATCH_PARAMETERS = (
40+
("if_etag_match", "If-Match"),
41+
("if_etag_not_match", "If-None-Match"),
42+
)
43+
3744
# generation match parameters in camel and snake cases
3845
_GENERATION_MATCH_PARAMETERS = (
3946
("if_generation_match", "ifGenerationMatch"),
@@ -147,6 +154,8 @@ def reload(
147154
self,
148155
client=None,
149156
projection="noAcl",
157+
if_etag_match=None,
158+
if_etag_not_match=None,
150159
if_generation_match=None,
151160
if_generation_not_match=None,
152161
if_metageneration_match=None,
@@ -168,6 +177,12 @@ def reload(
168177
Defaults to ``'noAcl'``. Specifies the set of
169178
properties to return.
170179
180+
:type if_etag_match: Union[str, Set[str]]
181+
:param if_etag_match: (Optional) See :ref:`using-if-etag-match`
182+
183+
:type if_etag_not_match: Union[str, Set[str]])
184+
:param if_etag_not_match: (Optional) See :ref:`using-if-etag-not-match`
185+
171186
:type if_generation_match: long
172187
:param if_generation_match:
173188
(Optional) See :ref:`using-if-generation-match`
@@ -205,10 +220,14 @@ def reload(
205220
if_metageneration_match=if_metageneration_match,
206221
if_metageneration_not_match=if_metageneration_not_match,
207222
)
223+
headers = self._encryption_headers()
224+
_add_etag_match_headers(
225+
headers, if_etag_match=if_etag_match, if_etag_not_match=if_etag_not_match
226+
)
208227
api_response = client._get_resource(
209228
self.path,
210229
query_params=query_params,
211-
headers=self._encryption_headers(),
230+
headers=headers,
212231
timeout=timeout,
213232
retry=retry,
214233
_target_object=self,
@@ -384,8 +403,7 @@ def update(
384403

385404

386405
def _scalar_property(fieldname):
387-
"""Create a property descriptor around the :class:`_PropertyMixin` helpers.
388-
"""
406+
"""Create a property descriptor around the :class:`_PropertyMixin` helpers."""
389407

390408
def _getter(self):
391409
"""Scalar property getter."""
@@ -449,6 +467,24 @@ def _convert_to_timestamp(value):
449467
return mtime
450468

451469

470+
def _add_etag_match_headers(headers, **match_parameters):
471+
"""Add generation match parameters into the given parameters list.
472+
473+
:type headers: dict
474+
:param headers: Headers dict.
475+
476+
:type match_parameters: dict
477+
:param match_parameters: if*etag*match parameters to add.
478+
"""
479+
for snakecase_name, header_name in _ETAG_MATCH_PARAMETERS:
480+
value = match_parameters.get(snakecase_name)
481+
482+
if value is not None:
483+
if isinstance(value, string_types):
484+
value = [value]
485+
headers[header_name] = ", ".join(value)
486+
487+
452488
def _add_generation_match_parameters(parameters, **match_parameters):
453489
"""Add generation match parameters into the given parameters list.
454490

‎google/cloud/storage/blob.py

Copy file name to clipboardExpand all lines: google/cloud/storage/blob.py
+78Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@
5959
from google.cloud._helpers import _rfc3339_nanos_to_datetime
6060
from google.cloud._helpers import _to_bytes
6161
from google.cloud.exceptions import NotFound
62+
from google.cloud.storage._helpers import _add_etag_match_headers
6263
from google.cloud.storage._helpers import _add_generation_match_parameters
6364
from google.cloud.storage._helpers import _PropertyMixin
6465
from google.cloud.storage._helpers import _scalar_property
@@ -634,6 +635,8 @@ def generate_signed_url(
634635
def exists(
635636
self,
636637
client=None,
638+
if_etag_match=None,
639+
if_etag_not_match=None,
637640
if_generation_match=None,
638641
if_generation_not_match=None,
639642
if_metageneration_match=None,
@@ -651,6 +654,14 @@ def exists(
651654
(Optional) The client to use. If not passed, falls back to the
652655
``client`` stored on the blob's bucket.
653656
657+
:type if_etag_match: Union[str, Set[str]]
658+
:param if_etag_match:
659+
(Optional) See :ref:`using-if-etag-match`
660+
661+
:type if_etag_not_match: Union[str, Set[str]]
662+
:param if_etag_not_match:
663+
(Optional) See :ref:`using-if-etag-not-match`
664+
654665
:type if_generation_match: long
655666
:param if_generation_match:
656667
(Optional) See :ref:`using-if-generation-match`
@@ -692,12 +703,19 @@ def exists(
692703
if_metageneration_match=if_metageneration_match,
693704
if_metageneration_not_match=if_metageneration_not_match,
694705
)
706+
707+
headers = {}
708+
_add_etag_match_headers(
709+
headers, if_etag_match=if_etag_match, if_etag_not_match=if_etag_not_match
710+
)
711+
695712
try:
696713
# We intentionally pass `_target_object=None` since fields=name
697714
# would limit the local properties.
698715
client._get_resource(
699716
self.path,
700717
query_params=query_params,
718+
headers=headers,
701719
timeout=timeout,
702720
retry=retry,
703721
_target_object=None,
@@ -1002,6 +1020,8 @@ def download_to_file(
10021020
start=None,
10031021
end=None,
10041022
raw_download=False,
1023+
if_etag_match=None,
1024+
if_etag_not_match=None,
10051025
if_generation_match=None,
10061026
if_generation_not_match=None,
10071027
if_metageneration_match=None,
@@ -1057,6 +1077,14 @@ def download_to_file(
10571077
:param raw_download:
10581078
(Optional) If true, download the object without any expansion.
10591079
1080+
:type if_etag_match: Union[str, Set[str]]
1081+
:param if_etag_match:
1082+
(Optional) See :ref:`using-if-etag-match`
1083+
1084+
:type if_etag_not_match: Union[str, Set[str]]
1085+
:param if_etag_not_match:
1086+
(Optional) See :ref:`using-if-etag-not-match`
1087+
10601088
:type if_generation_match: long
10611089
:param if_generation_match:
10621090
(Optional) See :ref:`using-if-generation-match`
@@ -1121,6 +1149,8 @@ def download_to_file(
11211149
start=start,
11221150
end=end,
11231151
raw_download=raw_download,
1152+
if_etag_match=if_etag_match,
1153+
if_etag_not_match=if_etag_not_match,
11241154
if_generation_match=if_generation_match,
11251155
if_generation_not_match=if_generation_not_match,
11261156
if_metageneration_match=if_metageneration_match,
@@ -1137,6 +1167,8 @@ def download_to_filename(
11371167
start=None,
11381168
end=None,
11391169
raw_download=False,
1170+
if_etag_match=None,
1171+
if_etag_not_match=None,
11401172
if_generation_match=None,
11411173
if_generation_not_match=None,
11421174
if_metageneration_match=None,
@@ -1168,6 +1200,14 @@ def download_to_filename(
11681200
:param raw_download:
11691201
(Optional) If true, download the object without any expansion.
11701202
1203+
:type if_etag_match: Union[str, Set[str]]
1204+
:param if_etag_match:
1205+
(Optional) See :ref:`using-if-etag-match`
1206+
1207+
:type if_etag_not_match: Union[str, Set[str]]
1208+
:param if_etag_not_match:
1209+
(Optional) See :ref:`using-if-etag-not-match`
1210+
11711211
:type if_generation_match: long
11721212
:param if_generation_match:
11731213
(Optional) See :ref:`using-if-generation-match`
@@ -1233,6 +1273,8 @@ def download_to_filename(
12331273
start=start,
12341274
end=end,
12351275
raw_download=raw_download,
1276+
if_etag_match=if_etag_match,
1277+
if_etag_not_match=if_etag_not_match,
12361278
if_generation_match=if_generation_match,
12371279
if_generation_not_match=if_generation_not_match,
12381280
if_metageneration_match=if_metageneration_match,
@@ -1260,6 +1302,8 @@ def download_as_bytes(
12601302
start=None,
12611303
end=None,
12621304
raw_download=False,
1305+
if_etag_match=None,
1306+
if_etag_not_match=None,
12631307
if_generation_match=None,
12641308
if_generation_not_match=None,
12651309
if_metageneration_match=None,
@@ -1288,6 +1332,14 @@ def download_as_bytes(
12881332
:param raw_download:
12891333
(Optional) If true, download the object without any expansion.
12901334
1335+
:type if_etag_match: Union[str, Set[str]]
1336+
:param if_etag_match:
1337+
(Optional) See :ref:`using-if-etag-match`
1338+
1339+
:type if_etag_not_match: Union[str, Set[str]]
1340+
:param if_etag_not_match:
1341+
(Optional) See :ref:`using-if-etag-not-match`
1342+
12911343
:type if_generation_match: long
12921344
:param if_generation_match:
12931345
(Optional) See :ref:`using-if-generation-match`
@@ -1355,6 +1407,8 @@ def download_as_bytes(
13551407
start=start,
13561408
end=end,
13571409
raw_download=raw_download,
1410+
if_etag_match=if_etag_match,
1411+
if_etag_not_match=if_etag_not_match,
13581412
if_generation_match=if_generation_match,
13591413
if_generation_not_match=if_generation_not_match,
13601414
if_metageneration_match=if_metageneration_match,
@@ -1371,6 +1425,8 @@ def download_as_string(
13711425
start=None,
13721426
end=None,
13731427
raw_download=False,
1428+
if_etag_match=None,
1429+
if_etag_not_match=None,
13741430
if_generation_match=None,
13751431
if_generation_not_match=None,
13761432
if_metageneration_match=None,
@@ -1401,6 +1457,14 @@ def download_as_string(
14011457
:param raw_download:
14021458
(Optional) If true, download the object without any expansion.
14031459
1460+
:type if_etag_match: Union[str, Set[str]]
1461+
:param if_etag_match:
1462+
(Optional) See :ref:`using-if-etag-match`
1463+
1464+
:type if_etag_not_match: Union[str, Set[str]]
1465+
:param if_etag_not_match:
1466+
(Optional) See :ref:`using-if-etag-not-match`
1467+
14041468
:type if_generation_match: long
14051469
:param if_generation_match:
14061470
(Optional) See :ref:`using-if-generation-match`
@@ -1460,6 +1524,8 @@ def download_as_string(
14601524
start=start,
14611525
end=end,
14621526
raw_download=raw_download,
1527+
if_etag_match=if_etag_match,
1528+
if_etag_not_match=if_etag_not_match,
14631529
if_generation_match=if_generation_match,
14641530
if_generation_not_match=if_generation_not_match,
14651531
if_metageneration_match=if_metageneration_match,
@@ -1475,6 +1541,8 @@ def download_as_text(
14751541
end=None,
14761542
raw_download=False,
14771543
encoding=None,
1544+
if_etag_match=None,
1545+
if_etag_not_match=None,
14781546
if_generation_match=None,
14791547
if_generation_not_match=None,
14801548
if_metageneration_match=None,
@@ -1507,6 +1575,14 @@ def download_as_text(
15071575
downloaded bytes. Defaults to the ``charset`` param of
15081576
attr:`content_type`, or else to "utf-8".
15091577
1578+
:type if_etag_match: Union[str, Set[str]]
1579+
:param if_etag_match:
1580+
(Optional) See :ref:`using-if-etag-match`
1581+
1582+
:type if_etag_not_match: Union[str, Set[str]]
1583+
:param if_etag_not_match:
1584+
(Optional) See :ref:`using-if-etag-not-match`
1585+
15101586
:type if_generation_match: long
15111587
:param if_generation_match:
15121588
(Optional) See :ref:`using-if-generation-match`
@@ -1558,6 +1634,8 @@ def download_as_text(
15581634
start=start,
15591635
end=end,
15601636
raw_download=raw_download,
1637+
if_etag_match=if_etag_match,
1638+
if_etag_not_match=if_etag_not_match,
15611639
if_generation_match=if_generation_match,
15621640
if_generation_not_match=if_generation_not_match,
15631641
if_metageneration_match=if_metageneration_match,

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.