Backfill partitioned Dags by partition-date range#67537
Merged
Conversation
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
5 times, most recently
from
44f05d4 to
c004a94
Compare
Lee-W
changed the title
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
2 times, most recently
from
0b1dcaa to
693aca6
Compare
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
from
693aca6 to
d510489
Compare
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
5 times, most recently
from
e3da6eb to
783d201
Compare
Lee-W
changed the title
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
3 times, most recently
from
2534af9 to
3391c38
Compare
Lee-W
requested review from
XD-DENG,
ashb,
bugraoz93,
dheerajturaga,
potiuk and
uranusjr
as code owners
uranusjr
reviewed
Jun 11, 2026
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
from
276d17c to
69fc2e2
Compare
uranusjr
reviewed
Jun 12, 2026
uranusjr
reviewed
Jun 12, 2026
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
from
69fc2e2 to
5e3d3c4
Compare
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
from
5e3d3c4 to
67f8049
Compare
uranusjr
reviewed
Jun 15, 2026
uranusjr
reviewed
Jun 15, 2026
* partitioned Dags backfill via --partition-date-start / --partition-date-end * non-partitioned Dags backfill via --from-date / --to-date * mixed or missing selectors raise dedicated ValueError subclasses Dates are compared as wall-clock dates in the timezone of the provided value. The REST API does not yet expose partition selectors.
…form The Run Backfill form now renders partition_date_start / partition_date_end for partitioned Dags (and from/to for the rest, switched on timetable_partitioned), and both the create and dry-run paths send the matching selectors so a partitioned-Dag backfill no longer fails. The partition window is validated start <= end on the client.
…o-date Backfilling a partitioned Dag now uses --from-date/--to-date, interpreted as the partition-date range when the Dag's timetable is partitioned. The separate --partition-date-start/--end flags, the mutual-exclusion validation, and the partition-specific API request and UI form fields are removed -- the partition vs date-range behavior is detected from the timetable, so one date range is passed regardless of Dag type.
… string partition_key is a formatted label; relabeling it (e.g. formatting in the timetable timezone) made backfill dedup miss already-scheduled runs and create duplicate partition runs on upgrade. Dedup now matches on partition_date — the format-independent UTC instant of the tick, shared by scheduled and backfill runs for the same partition. Adds a query-level and an end-to-end regression test covering a historical run keyed with the old UTC-instant label.
…tion Pin that asset-partitioned Dags (PartitionedAssetTimetable, periodic=False) are rejected from backfill and dry-run as non-periodic (HTTP 422) rather than reaching the partition iterator. Adds a param to the two existing non-periodic rejection tests.
Lee-W
force-pushed
the
dags-backfill-partition-range
branch
from
67f8049 to
2172d1a
Compare
uranusjr
removed request for
bbovenzi,
ephraimbuddy,
pierrejeambrun,
rawwar and
ryanahamilton
uranusjr
approved these changes
Jun 15, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Partition-based timetables advance along a partition-date axis rather than a logical-date interval, so the existing backfill path — which enumerates logical-date intervals — could not express "backfill these partitions". Rather than expose a separate set of partition flags, this teaches backfill to recognise a partitioned timetable and reuse the dates the user already passes, so there is nothing new to learn.
What
iter_partition_dagrun_infos(airflow-core/src/airflow/timetables/base.py): the baseTimetableraisesNotImplementedErrorby default;CronPartitionTimetable(timetables/trigger.py) implements it. Walking the partition-date axis, it yields oneDagRunInfoper cron tick in the half-open interval[earliest_partition_date, latest_partition_date)— withpartition_date=tick,partition_keyset to the tick's label in the timetable timezone,run_after=partition_date, anddata_interval=None._get_info_list(models/backfill.py) dispatches ondag.timetable.partitioned: partitioned Dags enumerate over[from_date.date(), to_date.date() + 1 day)viaiter_partition_dagrun_infos(resolve_day_boundsnaps to the local-midnight boundary; the+1 daymakes--to-dateinclusive); non-partitioned Dags keep the original logical-date interval path._format_keyformats the partition key in the timetable timezone, fixing cross-timezone labels — e.g. an Asia/Taipei midnight partition is labelled with its local date instead of the previous UTC day.--from-date/--to-dateare nowrequired=True, with help text noting that for a partitioned Dag this range is the partition-date range (detected automatically).closes: #65922
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the
guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.