Skip to content

airflow dags backfill: partition-range + ordering + concurrency #65922

Description

@Lee-W

Body

What

Extend airflow dags backfill for partitioned Dags so users can backfill a partition range with controlled ordering and concurrency. Basic partition backfill landed in #61464; what is missing is range flags (--partition-start / --partition-end), ordering control (in-order, parallel, pool-bounded waves), and concurrency caps independent of global parallelism. Example: deploy a new partitioned Dag and backfill 6
months of history in pool-bounded waves of 4.

Open question

Default ordering — chronological, parallel, or follow whatever the timetable suggests? How does the per-backfill concurrency cap interact with the Dag's max_active_runs and global parallelism?

Related

#65921

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

Labels

area:backfillSpecifically for backfill relatedkind:featureFeature Requestskind:metaHigh-level information important to the community

Type

No type
No fields configured for issues without a type.

Projects

Status
Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions