Skip to content

next run should show deps fulfillment e.g. 0 of 3 #25033

Description

@dstandish

on dags page (i.e. the home page) we have a "next run" column. for dataset-driven dags, since we can't know for certain when it will be, we could instead show how many deps are fulfilled, e.g. 0 of 1 and perhaps make it a link to the datasets that the dag is dependened on.

here's a sample query that returns the dags which are ready to run. but for this feature you'd need to get the num deps fulfilled and the total num deps.

        # these dag ids are triggered by datasets, and they are ready to go.
        dataset_triggered_dag_info_list = {
            x.dag_id: (x.first_event_time, x.last_event_time)
            for x in session.query(
                DatasetDagRef.dag_id,
                func.max(DDRQ.created_at).label('last_event_time'),
                func.max(DDRQ.created_at).label('first_event_time'),
            )
            .join(
                DDRQ,
                and_(
                    DDRQ.dataset_id == DatasetDagRef.dataset_id,
                    DDRQ.target_dag_id == DatasetDagRef.dag_id,
                ),
                isouter=True,
            )
            .group_by(DatasetDagRef.dag_id)
            .having(func.count() == func.sum(case((DDRQ.target_dag_id.is_not(None), 1), else_=0)))
            .all()
        }

Metadata

Metadata

Assignees

Labels

area:data-aware-schedulingassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitions

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions