chore(api)!: replace GraphQL observability API with gRPC, remove async-graphql dependencies by pront · Pull Request #24364 · vectordotdev/vector

pront · 2025-12-10T17:27:03Z

Summary

Migrates Vector's internal API from GraphQL to gRPC, significantly reducing code complexity while maintaining full backward compatibility for vector top and vector tap.

Removed (GraphQL)

Thousands of lines of GraphQL schema code were deleted 🎉
GraphQL dependencies: async-graphql, async-graphql-warp

Added (gRPC)

Protobuf schema: proto/vector/observability.proto
gRPC service implementation
gRPC client using tonic/prost (already existing dependencies)
Updated vector top and vector tap to use gRPC streaming

How did you test this PR?

cargo run -- top
cargo run -- tap
Updated and extended vector_api integration tests (component discovery, config reload, multi-output per-output metrics, tap across reload)
Existing unit tests

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

The comment now explicitly mentions that Log.metadata_full preserves event metadata during conversion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Migrate the vector_api integration tests from the deprecated GraphQL API to the new gRPC API. Changes: - Updated test infrastructure to use gRPC client instead of GraphQL - Migrated component queries from GraphQL to gRPC GetComponents RPC - Replaced GraphQL subscriptions with gRPC streaming for tap functionality - Added metrics population in gRPC GetComponents handler to support metrics tests - Fixed config reload tests to work around Vector behavior where components with unchanged names but modified connections don't appear during the reload transition - Increased startup timeout from 2s to 10s to handle slower test environments - Added 500ms delay before polling after SIGHUP to let Vector process the signal Test fixes: - Updated all assertions to use gRPC proto types (ComponentsResponse, ComponentType, etc.) - Changed tap tests to collect events inline instead of storing streams to avoid lifetime issues - Simplified reload tests to completely replace components instead of keeping old names - All 7 tests now pass: tap (4 tests), top (3 tests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Add breaking change changelog documenting the removal of the GraphQL Playground and replacement with gRPC API. Includes migration guide with grpcurl examples for common operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

changelog.d/graphql_to_grpc_api.breaking.md

- Add compatibility note to changelog: 0.55+ clients require 0.55+ server - Add grpc.io link to api.md - Link proto file directly in api.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fixes unused import/dead_code warnings when building without the api feature (e.g. component-validation-tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…etrics Consolidates StreamComponentReceivedEventsTotal, StreamComponentSentEventsTotal, StreamComponentReceivedBytesTotal, StreamComponentSentBytesTotal, StreamComponentErrorsTotal, StreamComponentReceivedEventsThroughput, StreamComponentSentEventsThroughput, StreamComponentReceivedBytesThroughput, and StreamComponentSentBytesThroughput into a single StreamComponentMetrics(ComponentMetricStreamRequest) RPC. The unified response type ComponentMetricResponse uses oneof value { TotalMetric total; ThroughputMetric throughput; } to carry per-output breakdowns alongside the aggregated value, keeping the schema honest about which fields are populated. StreamComponentAllocatedBytes remains a separate RPC since it is semantically distinct and has no per-output breakdown. Also moves ApiStarted internal event from grpc.rs (gated behind sources-vector/opentelemetry) into a new api.rs module gated behind the api feature flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

lib/vector-api-client/src/client.rs

lib/vector-top/src/state.rs

lib/vector-tap/Cargo.toml

website/content/en/docs/reference/api.md

…ust updates - Add METRIC_NAME_ prefix to MetricName enum values (ENUM_VALUE_PREFIX) - Rename proto package to vector.observability.v1 (PACKAGE_VERSION_SUFFIX) - Rename service Observability -> ObservabilityService (SERVICE_SUFFIX) - Rename all request/response messages to match their RPC names per RPC_REQUEST_RESPONSE_UNIQUE convention (e.g. MetaRequest -> GetMetaRequest, HeartbeatRequest -> StreamHeartbeatRequest, etc.) - Add COMPONENT_TYPE_UNSPECIFIED zero value to ComponentType enum (ENUM_ZERO_VALUE_SUFFIX) - Cascade all renames through Rust: service.rs, client.rs, metrics.rs, vector-tap, and integration tests - Update grpcurl examples in changelog Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…fix api.md title - Client::new now takes http::Uri instead of String; Endpoint is built eagerly so connect() can no longer fail with InvalidUrl - Remove unused `url` dep from vector-api-client, add `http` - Remove test_invalid_url and test_connection_failure (assert nothing useful) - Update all callers to parse their string URL into Uri - Fix api.md title to "The Vector Observability API" and update stale grpcurl examples to use vector.observability.v1.ObservabilityService Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move `TapRunner`, `EventFormatter`, `OutputChannel`, and `TapExecutorError` into a new `runner` module gated by the `api` feature, and make the associated heavy deps (`vector-api-client`, `prost`, `bytes`, `serde_*`, etc.) optional. Also removes the unused `tokio-tungstenite` dep. Restores `vector-lib/api = ["vector-tap/api"]` so the feature activation chain is preserved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move `#[allow(clippy::print_stderr)]` to the `fn cmd` level in both `src/top/cmd.rs` and `src/tap/cmd.rs` instead of per-statement blocks, and inline `{url}` format captures where applicable. Also applies rustfmt line-wrapping to service.rs and top.rs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The gRPC server already normalizes throughput values to per-second by dividing by `interval_secs` before streaming them. The `interval` field in the four `*Throughputs` EventType variants was therefore unused (`_interval`) and carried over from the old GraphQL client, which had to do the `v * (1000.0 / interval_ms)` normalization itself. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

src/tap/cmd.rs

src/top/cmd.rs

src/topology/controller.rs

src/api/grpc/service.rs

- Use OS entropy (thread RNG) instead of epoch-based seed for reservoir sampler RNG in service.rs; removes SystemTime/UNIX_EPOCH boilerplate - Return exitcode::USAGE (bad input) instead of exitcode::UNAVAILABLE when URL fails to parse in tap/top cmd - Rename tap_internal parameter initial_client -> mut client_opt, eliminating the intermediate local binding - Thread http::Uri through top::cmd -> subscription -> metrics::subscribe and metrics::init_components, removing the String -> Uri round-trip; re-export http::Uri from vector-api-client for callers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

thomasqueirozb

Very nice! Happy to see these abandoned graphql libraries being removed

changelog.d/graphql_to_grpc_api.breaking.md

…i-client Add http as a direct dep to vector-top and import http::Uri directly in metrics.rs and top/cmd.rs, rather than leaking it through a re-export in vector-api-client. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

datadog-vectordotdev · 2026-03-23T21:02:15Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a9c56a5 | Docs | Was this helpful? Give us feedback!}

The multi_output_transform_reports_per_output_sent_events test uses a route transform, but the feature was not included in vector-api-tests, causing Vector to exit with CONFIG error (78) when starting. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions bot added the domain: topology Anything related to Vector's topology code label Dec 10, 2025

pront mentioned this pull request Dec 12, 2025

Additional features for "vector top" #24337

Closed

pront force-pushed the pront-grpc branch from fd9c8b3 to 9eb5f0a Compare December 12, 2025 03:04

pront added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Dec 12, 2025

pront force-pushed the pront-grpc branch 7 times, most recently from a317991 to 45e8d0e Compare December 18, 2025 20:26

pront force-pushed the pront-grpc branch 3 times, most recently from d3b8168 to 5cec09b Compare December 22, 2025 19:02

pront force-pushed the pront-grpc branch 2 times, most recently from 216a7fc to 5b15df1 Compare January 9, 2026 23:50

esensar mentioned this pull request Jan 10, 2026

feat(cli): add scrolling, sorting and filtering to vector top #24355

Merged

8 tasks

pront force-pushed the pront-grpc branch from 5b15df1 to 7f7b736 Compare January 26, 2026 17:55

pront force-pushed the pront-grpc branch from 7f7b736 to 7eb90ea Compare February 6, 2026 23:25

pront mentioned this pull request Feb 12, 2026

chore(dev): add integration tests for the top command #24649

Merged

8 tasks

pront force-pushed the pront-grpc branch from 7eb90ea to de72c80 Compare February 25, 2026 21:18

chore(api): grpc server (remove graphql)

2e3ab45

pront force-pushed the pront-grpc branch from de72c80 to 2e3ab45 Compare February 25, 2026 21:30

pront and others added 5 commits February 25, 2026 18:03

reuse event.proto

7565d12

docs(api): clarify metadata preservation in proto conversion

cdfef02

The comment now explicitly mentions that Log.metadata_full preserves event metadata during conversion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

chore(dev): cargo fmt

3bebbd3

github-advanced-security AI found potential problems Feb 26, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into pront-grpc

7f875c2

pront changed the title ~~chore(api)!: grpc server (remove graphql)~~ chore(api): replace GraphQL observability API with gRPC, remove async-graphql dependencies Mar 19, 2026

pront and others added 2 commits March 19, 2026 16:06

docs: address thomasqueirozb review feedback

a2d43b4

- Add compatibility note to changelog: 0.55+ clients require 0.55+ server - Add grpc.io link to api.md - Link proto file directly in api.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(api): gate ApiStarted and its imports behind the api feature flag

e48f3e6

Fixes unused import/dead_code warnings when building without the api feature (e.g. component-validation-tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

thomasqueirozb changed the title ~~chore(api): replace GraphQL observability API with gRPC, remove async-graphql dependencies~~ chore(api)!: replace GraphQL observability API with gRPC, remove async-graphql dependencies Mar 19, 2026

thomasqueirozb reviewed Mar 20, 2026

View reviewed changes

pront and others added 6 commits March 20, 2026 14:31

style: collapse pub use statement in vector-tap lib.rs

0d43bc8

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

thomasqueirozb reviewed Mar 23, 2026

View reviewed changes

pront and others added 2 commits March 23, 2026 16:25

Merge remote-tracking branch 'origin/master' into pront-grpc

a96c4c7

thomasqueirozb approved these changes Mar 23, 2026

View reviewed changes

changelog.d/graphql_to_grpc_api.breaking.md Outdated Show resolved Hide resolved

pront and others added 2 commits March 23, 2026 16:38

docs: add link to proto file in changelog entry

5674215

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pront enabled auto-merge March 23, 2026 20:44

cargo fmt

011b8ec

pront added this pull request to the merge queue Mar 23, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 23, 2026

pront enabled auto-merge March 24, 2026 13:00

pront added this pull request to the merge queue Mar 24, 2026

Merged via the queue into master with commit 93a9771 Mar 24, 2026
59 checks passed

pront deleted the pront-grpc branch March 24, 2026 14:04

github-actions bot locked and limited conversation to collaborators Mar 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(api)!: replace GraphQL observability API with gRPC, remove async-graphql dependencies#24364

chore(api)!: replace GraphQL observability API with gRPC, remove async-graphql dependencies#24364
pront merged 81 commits intomasterfrom
pront-grpc

pront commented Dec 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb left a comment

Uh oh!

Uh oh!

datadog-vectordotdev bot commented Mar 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pront commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Removed (GraphQL)

Added (gRPC)

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

datadog-vectordotdev bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pront commented Dec 10, 2025 •

edited

Loading

datadog-vectordotdev bot commented Mar 23, 2026 •

edited

Loading