No way to supply custom annotations for cleanup cron job pods

[jonstacks]

Official Helm Chart version

1.4.0 (latest released)

Apache Airflow version

2.2.3 (latest released)

Kubernetes Version

v1.21.5

Helm Chart configuration

cleanup:
  enabled: true

airflowPodAnnotations:
    vault.hashicorp.com/agent-init-first: "true"
    vault.hashicorp.com/agent-inject: "false"
    vault.hashicorp.com/agent-run-as-user: "50000"
    vault.hashicorp.com/agent-pre-populate-only: "true"
    vault.hashicorp.com/agent-inject-status: "update"

Docker Image customisations

We have customized the ENTRYPOINT for exporting some environment variables that get loaded from Hashicorp's vault.

The entrypoint line in the Dockerfile:

ENTRYPOINT ["/usr/bin/dumb-init", "--", "/opt/airflow/entrypoint.sh"]

The last line in the /opt/airflow/entrypoint.sh script:

# Call Airflow's default entrypoint after we source the vault secrets
exec /entrypoint "${@}"

What happened

Install was successful and the webserver and scheduler pods are working as expected. The cleanup pods launched from the cleanup cronjob fail:

No vault secrets detected
....................
ERROR! Maximum number of retries (20) reached.

Last check result:
$ airflow db check
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 1129, in <module>
    conf.validate()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 224, in validate
    self._validate_config_dependencies()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 267, in _validate_config_dependencies
    raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the KubernetesExecutor

What you expected to happen

It looks like the annotations on the cleanup cronjob are static and only contain an istio annotation

airflow/chart/templates/cleanup/cleanup-cronjob.yaml

Lines 56 to 60 in c28c255

	{{- with .Values.labels }}
	{{ toYaml . | indent 12 }}
	{{- end }}
	annotations:
	sidecar.istio.io/inject: "false"

From the documentation in the values.yaml. I would expect the cleanup cronjob to have these annotations:

airflow/chart/values.yaml

Lines 187 to 189 in c28c255

	# Extra annotations to apply to all
	# Airflow pods
	airflowPodAnnotations: {}

How to reproduce

From the root of the airflow repository:

cd chart
helm dep build
helm template . --set cleanup.enabled=true --set airflowPodAnnotations."my\.test"="somevalue" -s templates/cleanup/cleanup-cronjob.yaml

If you look at the annotations section of the output, you will only see the static istio annotation that is hard coded.

Anything else

This could be a potentially breaking change even though the documentation says they should get applied to all Airflow pods. Another option would be to add cleanup.podAnnotations section for supplying them if fixing it by adding the global annotations would not work.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!