Improve guide for KubernetesPodOperator

[mik-laj]

Hello,

A better guide that describes how to use KubernetesPodOperator would be useful. This is one of the most frequently used Airflow operators, so many users have problems with this operator.
https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html
We currently have a guide that does not describe all the basic concepts. It is not very useful for the end user.

It would be fantastic if the following topics were discussed.

• How does KubernetesPodOperator work? Send request via Kubernetes API and waiting for the end. Airlfow uses labels to identity pod.
• How to use cluster ConfigMaps, Secrets, and Volumes with Pod?
• How to define env?
• How to use KubernetesPodOperaotr with YAML file/JSON Spec?
• How to define limit/requests resources?
• How to use KubernetesPodOperaotrs with Private Docker Image Registry?
• How does XCOM work and how to use it?
• How to debug problems with this operator? kubectl/kubernetes logs/kubernetes events
• How to use KubernetesPodOperator with GKE? GKEStartPodOperator
• What is the difference between KubernetesPodOperator and KubernetesExecutor?
• How to deal with common problems? FAQ section
• Best practice. e.g. separate node pool

If anyone is interested in this task, I am willing to provide all the necessary tips and information. If you are interested in writing a guide, you do not have to describe all the sections. Even one section will be very helpful and easier to learn about Airflow.

If anyone wants to learn more about this operator, I invite you to read the following articles.
https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator
https://www.astronomer.io/docs/kubepodoperator/
https://www.astronomer.io/docs/cli-kubepodoperator/
https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

Are you wondering how to start contributing to this project? Start by reading our contributor guide

[tanjinP]

I can take this on as it relates to #8883

[mik-laj]

@tanjinP Which section would you like to focus on at the beginning?

How does KubernetesPodOperator work? Send request via Kubernetes API and waiting for the end. Airlfow uses labels to identity pod.
How to define env/configmap/image pull secrets?
How to use KubernetesPodOperaotr with YAML file/JSON Spec?

What do you think about these 3 sections?

[tanjinP]

@mik-laj
I think we should include these as well - especially when we are using Kubernetes for the first time and if we want to pass around some data in the DAG context.

How to use KubernetesPodOperaotrs with Private Docker Image Registry?
How does XCOM work and how to use it?

[mik-laj]

@tanjinP Fantastic! I am waiting for your contribution.

[Siddharthk]

@tanjinP @mik-laj I was not able to find documentation on how to run airflow 1.10.10 on kubernetes. Can you guys help me point to that if yaml/helm charts already exists?

[mik-laj]

@Siddharthk https://airflow.readthedocs.io/en/latest/kubernetes.html
We have very limited documentation for Kubernetes. But it's worth you to look at #8777
https://github.com/apache/airflow-on-k8s-operator
We have also many good articles on the awesome list:
https://github.com/jghoman/awesome-apache-airflow

[Siddharthk]

@mik-laj thanks for the information. Eagerly waiting to try out the helm chart.