Skip to content

Add team_name tag to remaining multi-team metrics#69943

Open
Tegh25 wants to merge 1 commit into
apache:mainfrom
Tegh25:feature/add_team_name_tagging_dag_metrics
Open

Add team_name tag to remaining multi-team metrics#69943
Tegh25 wants to merge 1 commit into
apache:mainfrom
Tegh25:feature/add_team_name_tagging_dag_metrics

Conversation

@Tegh25

@Tegh25 Tegh25 commented Jul 16, 2026

Copy link
Copy Markdown

Should be able to close #68294 after merge.

Follow-up to #68367, #68589, #68593, #68594, #68599, #68108, and #68601.

Primary Changes

Added team_name tag to the following:
connection_test.reaped
connection_test.success
connection_test.failed
connection_test.hook_duration
scheduler.executor_heartbeat_duration
triggers.blocked_main_thread

Added a team_name field to TestConnection class. This is to share team_name from the scheduler to connection test supervisor without changing the execution API.

Excerpt from comment on closing issue:

The remaining scope was identified by @ferruzzi and is completed in this PR.

Remaining work (6)

Team-attributable, emit site has a team identity, but no team_name tag yet:

  • connection_test.reaped (scheduler reaper; ct.team_name is literally in scope and logged one line above the stats.incr, only prior_state is tagged)
  • connection_test.success (worker connection-test supervisor; per connection-test row)
  • connection_test.failed (same)
  • connection_test.hook_duration (same)
  • scheduler.executor_heartbeat_duration (loops over self.executors; each executor.team_name is available, tagged only by executor class)
  • triggers.blocked_main_thread (emitted in a team-scoped triggerer's TriggerRunner; team_name lives on the runner/supervisor but is not plumbed to this emit)

Two of these are more borderline than the others: scheduler.executor_heartbeat_duration
and triggers.blocked_main_thread are component-level (team-scoped executor /
team-scoped triggerer). They satisfy the signal list, but if those components are
considered "shared," they'd drop to non-team. The four connection_test.* rows are
clean Remaining items per the explicit "connection-test row" signal.


Was generative AI tooling used to co-author this PR?
  • Yes - Cursor (Auto)
    Generated-by: Cursor (Auto) following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:Executors-core LocalExecutor & SequentialExecutor area:providers area:Scheduler including HA (high availability) scheduler area:task-sdk area:Triggerer kind:documentation provider:edge Edge Executor / Worker (AIP-69) / edge3 labels Jul 16, 2026
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 18, 2026

@ferruzzi ferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks! I left two small notes before I'll approve. Those were solid catches on the out of date docs, but they shouldn't be lumped in with this change. Split them into a new PR and I'll approve both.

Comment thread airflow-core/docs/core-concepts/multi-team.rst Outdated
Comment thread airflow-core/docs/core-concepts/multi-team.rst Outdated
@Tegh25
Tegh25 force-pushed the feature/add_team_name_tagging_dag_metrics branch from badfdb0 to 9bcf752 Compare July 20, 2026 20:06
@Tegh25

Tegh25 commented Jul 20, 2026

Copy link
Copy Markdown
Author

Hi @ferruzzi, thank you. I reverted the doc changes and added them to a new PR at #70146.

@ferruzzi

ferruzzi commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Commented on the other PR; too far :P

The connection tests bullet and the scheduler.executor_heartbeat_duration / triggers.blocked_main_thread additions to the existing triggerer/executor bullets were directly related to this PR and should have stayed. The changes that were just "I noticed this while I was in here" should be moved out.

Sorry for the confusion.

Related to apache#68294

Added team_name tag to the following:
connection_test.reaped
connection_test.success
connection_test.failed
connection_test.hook_duration
scheduler.executor_heartbeat_duration
triggers.blocked_main_thread

Added a team_name field to TestConnection class. This is to share team_name from
the scheduler to connection test supervisor without changing the execution API.
@Tegh25
Tegh25 force-pushed the feature/add_team_name_tagging_dag_metrics branch from 9bcf752 to e272ab0 Compare July 20, 2026 22:22
@Tegh25
Tegh25 requested a review from ferruzzi July 20, 2026 22:24
@Tegh25

Tegh25 commented Jul 20, 2026

Copy link
Copy Markdown
Author

My bad. I amended the commit to include only the relevant doc changes, lmk what you think

Tegh25 added a commit to Tegh25/apache-airflow that referenced this pull request Jul 20, 2026
As discussed in PR apache#69943, these doc changes are being moved to
their own branch + PR instead of being attached to the other changes.
Tegh25 added a commit to Tegh25/apache-airflow that referenced this pull request Jul 20, 2026
As discussed in PR apache#69943, these doc changes are being moved to
their own branch + PR instead of being attached to the other changes.

@ferruzzi ferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI passing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Executors-core LocalExecutor & SequentialExecutor area:providers area:Scheduler including HA (high availability) scheduler area:task-sdk area:Triggerer kind:documentation provider:edge Edge Executor / Worker (AIP-69) / edge3 ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add team_name tagging to multi team dag metrics

3 participants