K8S Test: Set `[api/base_url]` with Helm in `Breeze`

[jason810496]

Body

Setting the [api/base_url] config dynamically in Kubernetes tests and performing a rolling restart of airflow-api-server slows down the tests.
Instead, we can set [api/base_url] to KUBERNETES_HOST_PORT when running breeze k8s deploy-airflow.

https://github.com/apache/airflow/blob/main/kubernetes_tests/test_base.py#L66-L67

---

For anyone working on this issue, you may find this commit helpful as a reference: 7bcf2e6.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[insomnes]

I would be happy to work on this one, if possible. Thank you!

[jason810496]

Sure, just assigned, thanks !

[insomnes]

@jason810496 while working on this one I've found that we also use default timers from helm chart for k8s checks on pods like this one on scheduler:

│     Startup:  exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \                                    │
│ airflow jobs check --job-type SchedulerJob --local                                                                                                 │
│ ] delay=0s timeout=20s period=10s #success=1 #failure=6

What do you think about tuning them? Would it be reasonable regarding the test suite speed up and k8s tests stability?
I think we can achieve minor gains on such checks by introducing a delay of 5 seconds and lowering the startup probe's period and timeout because they are quite high for the local kind cluster environment. And these timeouts could be configured the same way via helm.

I am also a bit puzzled by this test:

    @pytest.mark.parametrize("active_deadline_seconds", [10, 20])
    def test_kubernetes_pod_operator_active_deadline_seconds(self, active_deadline_seconds):

I think we can check it with lower timeouts and even only once, reason for the pod status would be enough. These two tests are the longest in the pod operator test suite.

Should I try to open separate PR on these to check?

[jason810496]

What do you think about tuning them? Would it be reasonable regarding the test suite speed up and k8s tests stability?
I think we can achieve minor gains on such checks by introducing a delay of 5 seconds and lowering the startup probe's period and timeout because they are quite high for the local kind cluster environment. And these timeouts could be configured the same way via helm.

Sure, that sounds good to me for system tests, as they can be configured via Helm options. Feel free to create another PR to fine-tune it!

These two tests are the longest in the pod operator test suite.

Actually, I haven't run the test_kubernetes_pod_operator_active_deadline_seconds locally before. Could the long duration of the test be related to cmds=["sh", "-c", "echo 'hello world' && sleep 60"], the command we use to test KubernetesPodOperator?
If so, maybe shortening the sleep time while still simulating a long-running job would be sufficient.

even only once

Not sure if running the test only once would be ideal.
cc @amoghrajesh

[insomnes]

Actually, I haven't run the test_kubernetes_pod_operator_active_deadline_seconds locally before. Could the long duration of the test be related to cmds=["sh", "-c", "echo 'hello world' && sleep 60"], the command we use to test KubernetesPodOperator?
If so, maybe shortening the sleep time while still simulating a long-running job would be sufficient.

As I understand the active_deadline_secods setting mechanism it sets the maximum running time for the pod (including PENDING state). So this test is checking cluster will fail the pod if it's running any longer. This way test duration depends on the timer set. But we also may lower the sleep value for speeding up the failure route of the case if it happens.

Not sure if running the test only once would be ideal.

It's a great idea to check with the test author! I didn't think about it, thanks!

I am sorry for the off-topic on this issue. I only wanted to pre-validate my ideas before trying the implementation. I hope I didn't bother you all much.

[jason810496]

As I understand the active_deadline_secods setting mechanism it sets the maximum running time for the pod (including PENDING state). So this test is checking cluster will fail the pod if it's running any longer. This way test duration depends on the timer set. But we also may lower the sleep value for speeding up the failure route of the case if it happens.

That sounds like a good approach to speeding up the test. Let's wait for the test author’s input.

It's a great idea to check with the test author! I didn't think about it, thanks!

I am sorry for the off-topic on this issue. I only wanted to pre-validate my ideas before trying the implementation. I hope I didn't bother you all much.

I think it's always beneficial to validate ideas before diving into implementation—it helps avoid unnecessary changes or refactoring. Feel free to discuss the details!

[insomnes]

I checked the test with lover active timeout values, and test timing definitely correlates with the setting value:

12.26s call     test_kubernetes_pod_operator.py::TestKubernetesPodOperator::test_kubernetes_pod_operator_active_deadline_seconds[3]
12.22s call     test_kubernetes_pod_operator.py::TestKubernetesPodOperator::test_kubernetes_pod_operator_active_deadline_seconds[5]