feat: add ability to retry query job automatically if the job fails #539
Description
Environment details
- OS type and version: Mac (10.15.7) and Python Base Docker Image
- Python version:
Python 3.7.8 - Poetry version:
Poetry version 1.0.10 google-cloud-bigquery:1.28.0 Google BigQuery API client library
Issue
I am using the retry parameter in biquery.Client.query but it isn't retrying my request.
Code example
Setup code:
import structlog
from google.api_core import retry, exceptions
from google.api_core.retry import if_exception_type, Retry
from google.cloud import bigquery
from common.logging.setup_logging import set_logger_config
set_logger_config() # Enables debug level logging for retry
project = "myproject"
source_table = f"{project}.mydataset.TABLE_NOT_REALLY_HERE"
statement = f"SELECT * FROM `{source_table}`"
client = bigquery.Client(project=project)Now if you run:
retry_policy = retry.Retry(predicate=if_exception_type(exceptions.NotFound))
query_job = client.query(statement, retry=retry_policy)
result = list(query_job.result(retry=retry_policy))
print(result)Only one request is made:
Traceback (most recent call last):
File "/Users/.../lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3230, in result
super(QueryJob, self).result(retry=retry, timeout=timeout)
File "/Users/.../lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 835, in result
return super(_AsyncJob, self).result(timeout=timeout)
File "/Users/.../lib/python3.7/site-packages/google/api_core/future/polling.py", line 134, in result
raise self._exception
google.api_core.exceptions.NotFound: 404 Not found: Table <TABLE_NAME> was not found in location US
(job ID: 5aeb5865-7bde-4795-ae80-75410f8b9282)
-----Query Job SQL Follows-----
| . | . | . | . | . | . | . | . | . | . | . |
1:SELECT *
2: FROM `<TABLE_NAME>`
3:
| . | . | . | . | . | . | . | . | . | . | . |
python-BaseException
Process finished with exit code 1
Workaround
If instead, I run:
@Retry(predicate=if_exception_type(exceptions.NotFound))
def do():
query_job = client.query(statement)
result = list(query_job.result())
print(result)
do()This works as expected and retries the request:
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:07:48.000317Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 1.0s ..."}
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:07:49.363842Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 3.2s ..."}
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:07:52.741042Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 2.8s ..."}
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:07:55.969272Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 9.2s ..."}
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:08:05.493544Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 4.2s ..."}
{"logger": "google.api_core.retry", "timestamp": "2021-03-02T19:08:09.960265Z", "severity": "debug", "message": "Retrying due to 404 Not found: Table <table_name> was not found in location US\n\n(job ID: <REMOVED>, sleeping 44.0s ..."}
Metadata
Metadata
Assignees
Labels
Issues related to the googleapis/python-bigquery API.Issues related to the googleapis/python-bigquery API.‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.