Skip to content

Fix custom timetables silently failing to schedule on Airflow 3.2 - #68326

Closed
Abdulrehman-PIAIC80387 wants to merge 3 commits into
apache:mainfrom
Abdulrehman-PIAIC80387:fix-custom-timetable-silent-scheduling-stop
Closed

Fix custom timetables silently failing to schedule on Airflow 3.2#68326
Abdulrehman-PIAIC80387 wants to merge 3 commits into
apache:mainfrom
Abdulrehman-PIAIC80387:fix-custom-timetable-silent-scheduling-stop

Conversation

@Abdulrehman-PIAIC80387

@Abdulrehman-PIAIC80387 Abdulrehman-PIAIC80387 commented Jun 10, 2026

Copy link
Copy Markdown

Airflow 3.2 extended DagRunInfo with two new fields for partition-oriented scheduling (AIP-76, #61167):

class DagRunInfo(NamedTuple):
    run_after: DateTime
    data_interval: DataInterval | None
    partition_date: DateTime | None   # new in 3.2
    partition_key: str | None         # new in 3.2

Both were added without defaults, making them required positional fields. Custom timetables written for 3.1.x and earlier — including the pattern shown in the official custom-timetable how-to — construct DagRunInfo with only two arguments:

return DagRunInfo(
    run_after=run_after,
    data_interval=DataInterval(start=start, end=end),
)

On 3.2 this raises TypeError: DagRunInfo.__new__() missing 2 required positional arguments: 'partition_date' and 'partition_key'. The exception is swallowed by the error handling around DAG.next_dagrun_info(), so next_dagrun / next_dagrun_create_after stay NULL, the Next Run column is empty, and no scheduled runs are created — with no import error or other visible failure. Built-in timetables are unaffected because they build DagRunInfo via helpers (DagRunInfo.interval(...), DagRunInfo.exact(...), etc.) that already pass the partition fields explicitly.

Fix

Default the two partition fields to None. They are the trailing fields of the NamedTuple, so the change is purely additive and restores the documented two-argument construction. Every internal call site already passes all four fields explicitly, so their behaviour is unchanged.

Design notes

  • Defaulting trailing NamedTuple fields is the idiomatic approach already used in this codebase — e.g. TaskInstanceKey (try_number: int = 1, map_index: int = -1). No __new__ override or dataclass conversion is needed.
  • Partitioned timetables continue to set partition_date / partition_key explicitly; the defaults only affect callers that omit them, which is exactly the pre-3.2 custom-timetable contract.

Tests

Added two unit tests in test_base_timetable.py: the backward-compatible two-argument construction (partition fields default to None) and the explicit four-argument construction (partition fields still settable). Full airflow-core/tests/unit/timetables/ suite passes (220 passed).

closes: #68315

@boring-cyborg

boring-cyborg Bot commented Jun 10, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Comment thread airflow-core/src/airflow/timetables/base.py Outdated
Comment thread airflow-core/src/airflow/timetables/base.py Outdated
Comment thread airflow-core/src/airflow/timetables/base.py Outdated
Comment thread airflow-core/tests/unit/timetables/test_base_timetable.py Outdated
@Abdulrehman-PIAIC80387
Abdulrehman-PIAIC80387 force-pushed the fix-custom-timetable-silent-scheduling-stop branch from cbdbaba to 47245e5 Compare June 12, 2026 06:20
@Abdulrehman-PIAIC80387

Abdulrehman-PIAIC80387 commented Jun 12, 2026

Copy link
Copy Markdown
Author

Hi @jroachgolf84, thanks for the review!

All comments are resolved in commit 47245e5. Any further feedback is welcome!

Comment thread airflow-core/tests/unit/timetables/test_base_timetable.py Outdated
Comment thread airflow-core/tests/unit/timetables/test_base_timetable.py Outdated
Comment thread airflow-core/tests/unit/timetables/test_base_timetable.py Outdated
@Lee-W

Lee-W commented Jun 12, 2026

Copy link
Copy Markdown
Member

#68342 is closer to being merged. So I'm going to merge that first. if you're still interested in it, we can extend the test case in the the parameterization way I mentioned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom timetable next_dagrun_info fails silently on 3.2 when DagRunInfo omits partition_date / partition_key

3 participants