add more information to PodLauncher timeout error

[jameslamb]

Thanks very much for all the work that has gone into v2 of KubernetesPodOperator!

This PR proposes two small changes (one for documentation, one to a log message in PodLauncher), which I think might help others in debugging task failures when using this operator.

Description

In my recent experience with KubernetesPodOperator (using airflow 2.1.0), I've found that for some classes of issues which cause a task to fail, it can be difficult to diagnose them from only the information in the Airflow UI.

For problems where kubernetes is able to create a pod but one or more of its containers fails to start, I've found that the task logs in the Airflow UI look something like this:

[WARN] Pod not yet started: some-pod-amhk4t
[WARN] Pod not yet started: some-pod-amhk4t
[WARN] Pod not yet started: some-pod-amhk4t
...
...
packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 131, in start_pod
    raise AirflowException("Pod took too long to start")
airflow.exceptions.AirflowException: Pod took too long to start

My first thought seeing that log message was "oh ok maybe image pulling is taking a while and I just need to set a higher timeout". My second thought was "ok..'too long' according to what configuration?".

I've found that the Airflow task logs can look like that for any of the following issues:

• referencing a secret that doesn't exist in the target namespace
• requesting an image that your pod isn't authorized to pull (e.g., it's in a private repository and you failed to specify imagePullSecrets)
• referencing a volume that does not exist

In these cases, diagnosing the issue requires going directly to kubernetes to get the pod events.

I hope that the changes in this PR might save others some debugging time in the future.

Thanks very much for your time and consideration.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[ephraimbuddy]

We should also add a test to ensure this exception message is not altered in the future accidentally

[jameslamb]

We should also add a test to ensure this exception message is not altered in the future accidentally

Ok sure, I can try to do that.

[jameslamb]

@ephraimbuddy ok, I think this is ready for review!

Sorry for the delay in responding to your request. I'm new to contributing to Airflow, so it took me a little while to figure out the test setup and how to mock my way into the point where the exception modified in this PR is raised.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[jedcunningham]

Thanks @jameslamb, congrats on your second commit 🎉