Open
Description
We're running a kubernetes cluster on Nebius and I'm using the Nebius CLI tool with a federated identity. However it seems like the ExecProvider of the Python kubernetes client is not working with this setup.
I've setup a kubeconfig file using nebius using:
nebius mk8s cluster get-credentials --id <id-redacted> --externalWhich resulted in a kubeconfig that looks like:
❯ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: <url-redacted>
name: <name-redacted>
contexts:
- context:
cluster: <cluster-redacted>
user: <user-redacted>
name: <name-redacted>
current-context: <context-redacted>
kind: Config
preferences: {}
users:
- name: <name-redacted>
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- mk8s
- v1
- cluster
- get-token
- --profile
- <user-redacted>
- --format
- json
command: /home/peter/.nebius/bin/nebius
env: null
interactiveMode: IfAvailable
provideClusterInfo: falseNote the exec provider section to get a key from nebius which works on it's own:
❯ nebius mk8s v1 cluster get-token --profile <user-redacted> --format json
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1beta1","spec":{"interactive":false},"status":{"expirationTimestamp":"2025-01-29T17:38:10Z","token":"<redacted>"}}And kubectl works:
❯ kubectl get namespaces
NAME STATUS AGE
...What happened (please include outputs or screenshots):
However the following fails:
from kubernetes import client, config
config.load_kube_config()
k8s_api = client.CoreV1Api()
k8s_api.api_client.configuration.debug = True
print(k8s_api.list_namespace())With error ERROR:root:exec: failed to decode process output: Expecting value: line 1 column 1 (char 0) and full output:
ERROR:root:exec: failed to decode process output: Expecting value: line 1 column 1 (char 0)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): <url-redacted>
DEBUG:urllib3.connectionpool:<url-redacted> "GET /api/v1/namespaces HTTP/1.1" 403 0
DEBUG:kubernetes.client.rest:response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}
send: b'GET /api/v1/namespaces HTTP/1.1\r\nHost: <url-redacted>\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nUser-Agent: OpenAPI-Generator/32.0.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Audit-Id: <id-redacted>
header: Cache-Control: no-cache, private
header: Content-Type: application/json
header: X-Content-Type-Options: nosniff
header: X-Kubernetes-Pf-Flowschema-Uid: <redacted>
header: X-Kubernetes-Pf-Prioritylevel-Uid: <redacted>
header: Date: Wed, 29 Jan 2025 08:47:02 GMT
header: Content-Length: 271
And this traceback (click to expand).
---------------------------------------------------------------------------
ApiException Traceback (most recent call last)
Cell In[2], line 6
4 k8s_api = client.CoreV1Api()
5 k8s_api.api_client.configuration.debug = True
----> 6 print(k8s_api.list_namespace())
File ~/miniforge3/envs/anam-audio-to-motion/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:14962, in CoreV1Api.list_namespace(self, **kwargs)
14930 """list_namespace # noqa: E501
14931
14932 list or watch objects of kind Namespace # noqa: E501
(...)
14959 returns the request thread.
14960 """
14961 kwargs['_return_http_data_only'] = True
> 14962 return self.list_namespace_with_http_info(**kwargs)
File lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:15073, in CoreV1Api.list_namespace_with_http_info(self, **kwargs)
15070 # Authentication setting
15071 auth_settings = ['BearerToken'] # noqa: E501
> 15073 return self.api_client.call_api(
15074 '/api/v1/namespaces', 'GET',
15075 path_params,
15076 query_params,
15077 header_params,
15078 body=body_params,
15079 post_params=form_params,
15080 files=local_var_files,
15081 response_type='V1NamespaceList', # noqa: E501
15082 auth_settings=auth_settings,
15083 async_req=local_var_params.get('async_req'),
15084 _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501
15085 _preload_content=local_var_params.get('_preload_content', True),
15086 _request_timeout=local_var_params.get('_request_timeout'),
15087 collection_formats=collection_formats)
File lib/python3.11/site-packages/kubernetes/client/api_client.py:348, in ApiClient.call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
311 """Makes the HTTP request (synchronous) and returns deserialized data.
312
313 To make an async_req request, set the async_req parameter.
(...)
345 then the method will return the response directly.
346 """
347 if not async_req:
--> 348 return self.__call_api(resource_path, method,
349 path_params, query_params, header_params,
350 body, post_params, files,
351 response_type, auth_settings,
352 _return_http_data_only, collection_formats,
353 _preload_content, _request_timeout, _host)
355 return self.pool.apply_async(self.__call_api, (resource_path,
356 method, path_params,
357 query_params,
(...)
365 _request_timeout,
366 _host))
File lib/python3.11/site-packages/kubernetes/client/api_client.py:180, in ApiClient.__call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
177 url = _host + resource_path
179 # perform request and return response
--> 180 response_data = self.request(
181 method, url, query_params=query_params, headers=header_params,
182 post_params=post_params, body=body,
183 _preload_content=_preload_content,
184 _request_timeout=_request_timeout)
186 self.last_response = response_data
188 return_data = response_data
File lib/python3.11/site-packages/kubernetes/client/api_client.py:373, in ApiClient.request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
371 """Makes the HTTP request using RESTClient."""
372 if method == "GET":
--> 373 return self.rest_client.GET(url,
374 query_params=query_params,
375 _preload_content=_preload_content,
376 _request_timeout=_request_timeout,
377 headers=headers)
378 elif method == "HEAD":
379 return self.rest_client.HEAD(url,
380 query_params=query_params,
381 _preload_content=_preload_content,
382 _request_timeout=_request_timeout,
383 headers=headers)
File lib/python3.11/site-packages/kubernetes/client/rest.py:244, in RESTClientObject.GET(self, url, headers, query_params, _preload_content, _request_timeout)
242 def GET(self, url, headers=None, query_params=None, _preload_content=True,
243 _request_timeout=None):
--> 244 return self.request("GET", url,
245 headers=headers,
246 _preload_content=_preload_content,
247 _request_timeout=_request_timeout,
248 query_params=query_params)
File lib/python3.11/site-packages/kubernetes/client/rest.py:238, in RESTClientObject.request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
235 logger.debug("response body: %s", r.data)
237 if not 200 <= r.status <= 299:
--> 238 raise ApiException(http_resp=r)
240 return r
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '<redacted>', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': ''<redacted>', 'X-Kubernetes-Pf-Prioritylevel-Uid': ''<redacted>', 'Date': 'Wed, 29 Jan 2025 08:47:02 GMT', 'Content-Length': '271'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}Environment:
❯ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.30.7
WARNING: version difference between client (1.32) and server (1.30) exceeds the supported minor version skew of +/-1
❯ python --version
Python 3.11.11
❯ pip list | grep kubernetes
kubernetes 32.0.0
❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
Metadata
Metadata
Assignees
Labels
Categorizes issue or PR as related to a bug.Categorizes issue or PR as related to a bug.Denotes an issue or PR has remained open with no activity and has become stale.Denotes an issue or PR has remained open with no activity and has become stale.