KubernetesExecutor does not work with exec authentication on kubernetes client >= 22.6.0

[dszakallas]

Apache Airflow version

2.2.4 (latest released)

What happened

When using KubernetesExecutor with in_cluster = False in conjunction with a Config with an exec user using kubelogin, something wrong happens and authentication does not take place:

hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 747, in _adopt_completed_pods
hack-scheduler-1  |     pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
hack-scheduler-1  |     return self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
hack-scheduler-1  |     return self.api_client.call_api(
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
hack-scheduler-1  |     return self.__call_api(resource_path, method,
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
hack-scheduler-1  |     response_data = self.request(
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
hack-scheduler-1  |     return self.rest_client.GET(url,
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
hack-scheduler-1  |     return self.request("GET", url,
hack-scheduler-1  |   File "/airflow/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request
hack-scheduler-1  |     raise ApiException(http_resp=r)
hack-scheduler-1  | kubernetes.client.exceptions.ApiException: (401)
hack-scheduler-1  | Reason: Unauthorized
hack-scheduler-1  | HTTP response headers: HTTPHeaderDict({'Audit-Id': '9d09a92f-d294-4a82-9aac-bbafe9573469', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sun, 27 Mar 2022 21:16:39 GMT', 'Content-Length': '129'})

I managed to trace back to the source of the error to RefreshConfiguration, and create a workaround.

airflow/airflow/kubernetes/kube_client.py

Lines 45 to 46 in ee9049c

	cfg = RefreshConfiguration()
	load_kube_config(client_configuration=cfg, config_file=config_file, context=cluster_context)

Bypassing the RefreshConfiguration by changing the above two lines to:

config.load_kube_config(context=cluster_context, config_file=configfile)
cfg = None

resolves the problem. I am still debugging what exactly the problem is with RefreshConfiguration and kubelogin.

Factoids:

• I am facing this issue with the service principal authentication flow with AKS cluster. I didn't try other flows so they might or might not work.
• I am using kubelogin v0.0.11

What you think should happen instead

Authentication should work without a problem.

How to reproduce

It's hard to reproduce given the specificity of the problem.

1. Create a service principal and assign permissions to be able to create resources on the AKS cluster.
2. Install kubelogin
3. Create a Config file that uses the kubelogin exec authentication flow with service principal authentication with the correct values filled in. See docs
4. Confirm it works by running

   from kubernetes import client, config
   config.load_kube_config()
   print(client.CoreV1Api().list_namespaced_pod('default'))

5. Try out with Airflow and get lots of 401 errors.

Operating System

Debian

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Proof of concept deployment with Docker compose for local development purposes using KubernetesExecutor to schedule worker pods in an AKS cluster.

Anything else

This issue happens every time.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[dszakallas]

I did a bit more of debugging, it seems that the error is not specific to kubelogin and a much more general problem. I am renaming the ticket as it has little to do with kubelogin and Azure.

I found this in the logs:

{{refresh_config.py:73}} ERROR - __init__() missing 1 required positional argument: 'cwd'

I am using the following version of kubernetes client:

kubernetes                               23.3.0

Here's the relevant diff in the upstream:

kubernetes-client/python@1c5bf58

So it seems like this API breaks in v22.6.0.

Changing

airflow/airflow/kubernetes/refresh_config.py

Line 61 in ee9049c

status = ExecProvider(self._user['exec']).run()

to

            status = ExecProvider(self._user['exec'], os.getcwd()).run()

resolves the issue.

[eladkal]

@dszakallas can you open PR with the suggested fix?

[potiuk]

That might be another one (@ephraimbuddy @dstandish) that might make 2.2.5 needing an RC2
There is this huge bump of K8S client version from 2.2.4 (11.0.0 -> 23.3.0) so I think we should be extra careful and possibly do some more testing to prevent regressions there, otherwise we might need to release 2.2.6 reallly quickly.

[jedcunningham]

@dszakallas, not sure what version of the kubernetes provider you have, but try with this combo:

kubernetes==11.0.0
apache-airflow-providers-cncf-kubernetes==3.1.1

Basically, 2.2.4 only works with 11.0.0, and provider 3.1.2 only works with k8s >=21.7.0. The "common ground" is the above versions.

[jedcunningham]

I'll also add, you should be using constraints, which gets you on versions all tested together:
https://github.com/apache/airflow/blob/constraints-2.2.4/constraints-3.7.txt
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files

That would have gotten you 11.0.0 and 3.0.2 which would work together as well.

[dstandish]

closing for the following reasons

this issue should be resolved by using constraints file as suggested by @jedcunningham

And from airflow 2.3.0, we remove anyway the “refresh configuration” code that is the problem here.