Intermittent Cloud Datastore Error: Unauthorized: 401 ... Unable to find the server at accounts.google.com #3085
Description
I am getting regular, but intermittent errors when using GCL to read from the datastore. The stacktrace is below. I am running Python 2.7 on Ubuntu.
I get this error only occasionally when I call the get_multi function. It is intermittent and looks like it could be a network error based on the very last line (Unable to find the server at accounts.google.com.). I am making thousands of calls to the datastore, and while this occurs only a small percent of the time, it occurs with pretty heavy regularity (about once a minute).
File "/home/speedplane/DA/djangoappengine/db_gcl/compiler.py", line 349, in get_matching_pk
self.client.get_multi(self.included_pks, **config)
File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/client.py", line 307, in get_multi
transaction_id=transaction and transaction.id,
File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/client.py", line 124, in _extended_lookup
transaction_id=transaction_id,
File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 519, in lookup
lookup_response = self._datastore_api.lookup(project, lookup_request)
File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 304, in lookup
return self._stub.Lookup(request_pb)
File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 260, in _grpc_catch_rendezvous
raise error_class(exc.details())
Unauthorized: 401 Traceback (most recent call last):. File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6918). File "/usr/local/lib/python2.7/dist-packages/grpc/_plugin_wrapping.py", line 107, in __call__. AuthMetadataPluginCallback(wrapped_cygrpc_callback)). File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 65, in __call__. callback(self._get_authorization_headers(), None). File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 51, in _get_authorization_headers. self._credentials.refresh(self._request). File "/usr/local/lib/python2.7/dist-packages/google/oauth2/service_account.py", line 313, in refresh. request, self._token_uri, assertion). File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant. response_data = _token_endpoint_request(request, token_uri, body). File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request. method='POST', url=token_uri, headers=headers, body=body). File "/usr/local/lib/python2.7/dist-packages/google_auth_httplib2.py", line 119, in __call__. raise exceptions.TransportError(exc).TransportError: Unable to find the server at accounts.google.com.
That last line in the stacktrace above has its own stacktrace built-in. I've expanded it below for ease of reading:
Unauthorized: 401 Traceback (most recent call last):.
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6918).
File "/usr/local/lib/python2.7/dist-packages/grpc/_plugin_wrapping.py", line 107, in __call__.
AuthMetadataPluginCallback(wrapped_cygrpc_callback)).
File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 65, in __call__.
callback(self._get_authorization_headers(), None).
File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 51, in _get_authorization_headers.
self._credentials.refresh(self._request).
File "/usr/local/lib/python2.7/dist-packages/google/oauth2/service_account.py", line 313, in refresh.
request, self._token_uri, assertion).
File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant.
response_data = _token_endpoint_request(request, token_uri, body).
File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request.
method='POST', url=token_uri, headers=headers, body=body).
File "/usr/local/lib/python2.7/dist-packages/google_auth_httplib2.py", line 119, in __call__.
raise exceptions.TransportError(exc).TransportError: Unable to find the server at accounts.google.com.
I am running this within a Docker container orchestrated by Kubernetes. My Dockerfile inherits from ubuntu:trusty. Below is version information:
# lsb_release
LSB Version: core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
# python --version
Python 2.7.6
naoko and ravindranathakila
Metadata
Metadata
Assignees
Labels
Issues related to the Datastore API.Issues related to the Datastore API.