K8s API requests should be retried in async KPO trigger

[dstandish]

Body

We don't retry them so, as soon as there is one failure, the trigger fails, and the task fails immediately.

We should definitely implement retries, perhaps backoff of some kind.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[insomnes]

I would be happy to work on this, if possible. Thank you!

[dstandish]

Great @insomnes . Could you also please ensure that we make clear, when we raise due to trigger error, that we indicate that. Right now the user is left to sort of figure that out. Probably a log.error("trigger emitted a failure event; failing task") right before the raise would be sufficient.

[insomnes]

Thank you!

Could you also please ensure that we make clear, when we raise due to trigger error, that we indicate that.

Duly noted

[insomnes]

@dstandish hello! Sorry for the silence. I had a problem with my dev setup due to the provider migration. During that, I had a small draft and was researching how things are done outside the trigger and in sync hook.
I think I would be able to finalize my draft this week with auto-tests and manual checks.