Skip to content

Fix Celery worker crash on Airflow 3.0 with json_logs#69919

Merged
vatsrahul1001 merged 1 commit into
mainfrom
fix-celery-json-output-af30
Jul 16, 2026
Merged

Fix Celery worker crash on Airflow 3.0 with json_logs#69919
vatsrahul1001 merged 1 commit into
mainfrom
fix-celery-json-output-af30

Conversation

@vatsrahul1001

@vatsrahul1001 vatsrahul1001 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

The Celery worker CLI passes json_output to airflow.sdk.log.configure_logging under an AIRFLOW_V_3_0_PLUS gate (added in #68916 "Honor json_logs config in Celery worker startup"):

if AIRFLOW_V_3_0_PLUS:
    from airflow.sdk.log import configure_logging
    ...
    configure_logging(output=sys.stdout.buffer, json_output=json_output)

But json_output was only added to the Task SDK's configure_logging in SDK 1.1 / Airflow 3.1 (the structlog migration, #52651). On Airflow 3.0.x that parameter does not exist, so the call raises:

TypeError: configure_logging() got an unexpected keyword argument 'json_output'

…and the Celery worker crashes on startup. The AIRFLOW_V_3_0_PLUS gate is too loose — this affects any plain Airflow 3.0.x deployment running celery provider >= 3.22.0.

How

  • Add an AIRFLOW_V_3_1_PLUS constant to the provider's version_compat.
  • Gate the json_logs handling on AIRFLOW_V_3_1_PLUS. On Airflow 3.0.x, fall back to configure_logging(output=sys.stdout.buffer) (no json_output), matching the pre-Honor json_logs config in Celery worker startup #68916 behavior.

This keeps the provider compatible across the full Airflow 3.x range it declares support for; json_logs is honored from 3.1+ where the SDK supports it.

Tests

Added test_json_output_not_passed_on_airflow_3_0 (patches AIRFLOW_V_3_1_PLUS = False, asserts configure_logging is called without json_output). Existing json_logs tests continue to assert the 3.1+ behavior.


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code (Opus 4.8) following the guidelines

I reviewed and understand all of the generated code, verified it against the actual worker() code path, and ran the relevant static checks and tests locally.

@kaxil kaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Correct version boundary -- verified against stable-branch content that json_output lands in the task-sdk configure_logging on 3.1+, so gating the 3.0.x fallback on AIRFLOW_V_3_1_PLUS is right. Fix restores pre-#68916 behavior and the regression test has teeth.

The celery worker CLI unconditionally passed `json_output` to
`airflow.sdk.log.configure_logging` under an `AIRFLOW_V_3_0_PLUS` gate.
That parameter was only added to the Task SDK in 1.1 (Airflow 3.1, the
structlog migration), so on Airflow 3.0.x the call raises
`TypeError: configure_logging() got an unexpected keyword argument
'json_output'` and the worker crashes on startup.

Gate the `json_logs` handling on a new `AIRFLOW_V_3_1_PLUS` constant and
fall back to `configure_logging(output=sys.stdout.buffer)` on 3.0.x, so
the provider stays compatible across the full Airflow 3.x range it
supports.

Signed-off-by: Rahul Vats <rah.sharma11@gmail.com>
@vatsrahul1001
vatsrahul1001 force-pushed the fix-celery-json-output-af30 branch from f1339dc to 6adb60c Compare July 15, 2026 14:31
@vatsrahul1001
vatsrahul1001 merged commit 10e310e into main Jul 16, 2026
79 checks passed
@vatsrahul1001
vatsrahul1001 deleted the fix-celery-json-output-af30 branch July 16, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants