Skip to content

feat: Add control plane telemetry - actor crashes counter and operation labels - #642

Open
Angela (Angelawork) wants to merge 6 commits into
agent-substrate:mainfrom
Angelawork:feature/actor-crashes-telemetry
Open

feat: Add control plane telemetry - actor crashes counter and operation labels#642
Angela (Angelawork) wants to merge 6 commits into
agent-substrate:mainfrom
Angelawork:feature/actor-crashes-telemetry

Conversation

@Angelawork

Copy link
Copy Markdown
Collaborator

Fixes #564 (Part 2)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements the second part of #564 by adding telemetry counter instrumentation for ate.actor.crashes in ateapi and establishing bounded label validation for operations and crash failure causes.

Key Changes:

  • Updated maybeCrashActor and crashActor to reference ateattr.OperationName* constants and normalize opName.
  • Passed explicit failure reasons (ReasonCorruptedAssignment, ReasonWorkerPodGone, ReasonWorkerReassigned) across workflow_resume.go, workflow_suspend.go, and workflow_pause.go.
  • Instrumented WorkerPoolSyncer.releaseActorOnDeadWorker in syncer.go to record recordActorCrash when background pod deletion crashes an actor.
  • Instrumented FinalizePausedStep in workflow_pause.go to record recordActorCrash when nodeName is missing during pause finalization.
  • Added ate_actor_crashes to PlatformMetricPrefixes in collector_metrics.go.
  • Added pre-creation resource cleanup in metrics_test.go to prevent AlreadyExists errors on test reruns.
  • Updated metrics_test.go to trigger an actor crash via UpdateActor and ResumeActor, asserting ate_actor_crashes emission and label presence in OTel collector scrape outputs.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork Angela (Angelawork) changed the title Feature/actor crashes telemetry feat: extend control plane telemetry with actor crashes counter and operation labels Jul 30, 2026
@Angelawork Angela (Angelawork) changed the title feat: extend control plane telemetry with actor crashes counter and operation labels feat: Add control plane telemetry - actor crashes counter and operation labels Jul 30, 2026
@Angelawork
Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 2 times, most recently from 4099ed0 to 0e45f4f Compare July 30, 2026 20:24
@Angelawork
Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from 0e45f4f to 65c4522 Compare July 30, 2026 20:45
Comment thread internal/ateattr/ateattr.go Outdated
Comment thread cmd/ateapi/internal/controlapi/crash.go Outdated
Comment thread cmd/ateapi/internal/controlapi/crash.go Outdated
Comment thread cmd/ateapi/internal/controlapi/crash.go Outdated
Comment thread internal/ateerrors/ateerrors.go
Comment thread cmd/ateapi/internal/controlapi/syncer.go Outdated
Comment thread internal/ateattr/ateattr.go Outdated
Comment thread internal/e2e/suites/metrics/metrics_test.go Outdated
Comment thread docs/observability.md Outdated
Comment thread docs/observability.md Outdated
Comment thread internal/e2e/suites/metrics/metrics_test.go
Comment thread internal/e2e/suites/metrics/metrics_test.go
Comment thread internal/ateattr/ateattr.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from fa2c10c to f69db52 Compare July 31, 2026 16:56
Comment thread cmd/ateapi/internal/controlapi/crash_test.go Outdated
Comment thread cmd/ateapi/internal/controlapi/syncer.go Outdated
Comment thread cmd/ateapi/internal/controlapi/workflow_pause.go Outdated
Comment thread cmd/ateapi/internal/controlapi/metrics.go Outdated
Comment thread cmd/ateapi/internal/controlapi/crash.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from b19d783 to ab0a9bc Compare July 31, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

3 participants