feat(datafusion): DataFusion metrics query layer by alexanderbianchi · Pull Request #6276 · quickwit-oss/quickwit

alexanderbianchi · 2026-04-07T16:46:47Z

Adds a DataFusion-based query execution layer for OSS parquet metrics. Review commit by commit. Everything is disabled by default.

Commits

f8de0eca — quickwit-datafusion core framework

QuickwitDataSource trait: plugin point for data sources (contributions, create_default_table_provider, list_index_names, optional init/register_for_worker/try_consume_read_rel hooks)
DataSourceContributions: additive value type for optimizer rules, UDFs, codec appliers; check_invariants detects name collisions at startup
QuickwitSchemaProvider: routes table resolution to DDL-registered tables (via MemorySchemaProvider) then source create_default_table_provider
DataFusionSessionBuilder: shared Arc<RuntimeEnv> across sessions; installs DistributedPhysicalOptimizerRule + QuickwitTaskEstimator when a resolver is present
build_quickwit_worker / QuickwitWorkerSessionBuilder: shares coordinator RuntimeEnv so registered object stores are visible on workers without re-registration
QuickwitWorkerResolver: maps SearcherPool addresses to worker gRPC URLs
QuickwitObjectStore: read-only quickwit_storage::Storage → object_store::ObjectStore bridge
DataFusionService: streaming SQL execution; handles semicolon-separated DDL+query batches

bce31c87 — MetricsDataSource

MetricsDataSource implements QuickwitDataSource; delegates to MetricsIndexResolver, registers object stores on workers concurrently
MetricsTableProvider: ParquetSource-backed scan with bloom filter, page-index, and filter pushdown enabled; one file group per split
Predicate extraction (predicate.rs): extracts MetricsSplitQuery from DataFusion filters; unwraps CAST/TryCast nodes so time-range predicates survive DataFusion's type-coercion pass
MetastoreSplitProvider: multi-value IN lists on tag columns are not forwarded to the metastore (single Option<String> per field); parquet-level filter handles correctness
MetastoreIndexResolver: index_metadata RPC + StorageResolver::resolve; no cache (follow-up)
MetricsTableProviderFactory: STORED AS metrics DDL support

8359a4a4 — QuickwitSubstraitConsumer

Implements SubstraitConsumer from datafusion-substrait; routes each ReadRel to registered sources via try_consume_read_rel, falls back to catalog
WithCustomProvider shim: injects a pre-built provider for one table without rebuilding the session
Rewrites ExtensionTable rels to NamedTable after a source claims them so from_read_rel applies standard filter/projection handling
execute_substrait_plan_streaming takes full SessionContext (not SessionState) so catalog registrations from build_session remain visible
Adds execute_substrait and execute_substrait_json to DataFusionService

feaceece7 — gRPC wiring + integration tests

datafusion.proto: DataFusionService with ExecuteSubstrait and ExecuteSql server-streaming RPCs; each response carries one RecordBatch as Arrow IPC bytes
DataFusionServiceGrpcImpl: tonic adapter streaming via mpsc + ReceiverStream; InvalidArgument for plan/schema errors, Internal otherwise
Mounted in quickwit-serve when datafusion_session_builder is Some; gRPC reflection registered only when enabled
Integration tests: SQL pruning, aggregation, time-range filtering, GROUP BY, distributed tasks (PartitionIsolatorExec, no shuffles), NULL fill for missing parquet columns, Substrait named-table queries, rollup from JSON plan

🤖 Generated with Claude Code

Introduces a generic DataFusion execution layer with a pluggable QuickwitDataSource trait. No data-source-specific code. - QuickwitDataSource trait + DataSourceContributions (contribution-return pattern) - DataFusionSessionBuilder with shared RuntimeEnv, check_invariants - QuickwitSchemaProvider backed by DataFusion MemorySchemaProvider for DDL tables - QuickwitWorkerSessionBuilder + build_quickwit_worker for distributed execution - QuickwitWorkerResolver, QuickwitTaskEstimator - QuickwitObjectStore: quickwit_storage::Storage → object_store::ObjectStore bridge - DataFusionService::execute_sql (streaming Arrow IPC responses) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Implements QuickwitDataSource for the parquet metrics pipeline from PR quickwit-oss#6237. - MetricsDataSource: production (metastore-backed) and test (SimpleIndexResolver) - MetricsTableProvider: filter pushdown with CAST-unwrapping fix for timestamp - MetastoreSplitProvider: converts MetricsSplitQuery → ListMetricsSplitsQuery - MetastoreIndexResolver: resolves index URI → QuickwitObjectStore per query - MetricsSplitQuery + extract_split_filters: predicate extraction for split pruning - MetricsTableProviderFactory: CREATE EXTERNAL TABLE … STORED AS metrics support - test_utils: make_batch, TestSplitProvider, MetricsTestbed for integration tests Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Routes Substrait ReadRel nodes to registered QuickwitDataSource implementations. Standard NamedTable reads resolve via MetricsDataSource::try_consume_read_rel. ExtensionTable reads (custom protos) can be handled by downstream callers. - QuickwitSubstraitConsumer implementing datafusion-substrait SubstraitConsumer - execute_substrait_plan / execute_substrait_plan_streaming entry points - DataFusionService::execute_substrait (bytes) and execute_substrait_json (dev path) - session.rs: DataFusionSessionBuilder::execute_substrait convenience method Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…tegration tests - Add datafusion.proto with DataFusionService (ExecuteSubstrait, ExecuteSql RPCs) - Generate codegen and mod.rs for the new proto service - Wire DataFusionService and WorkerService into quickwit-serve gRPC layer - Add DataFusionServiceGrpcImpl handler - Auto-create otel-metrics-v0_9 index on startup alongside logs/traces - Add metrics_datafusion_tests: in-process SQL + Substrait over parquet splits - Add metrics_distributed_tests: multi-node distributed execution - Add rollup_substrait.json fixture for Substrait plan testing Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Remove extra schema argument from ParquetWriter::new; the API only accepts a ParquetWriterConfig. Remove unused ParquetSchema import. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…test - Remove erroneous ParquetSchema argument from ParquetWriter::new calls in integration tests (API takes only ParquetWriterConfig) - Mark test_rest_ingest_then_in_process_query as #[ignore] until the /ingest-metrics REST endpoint is wired in quickwit-serve Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

alexanderbianchi · 2026-04-08T20:53:23Z

@mattmkim left some comments on this draft PR that I closed #6270 , addressed in 07f8548

mattmkim

disclaimer only really reviewed the metrics specific logic, not a DF expert. made comments here,

what would give us more confidence in merging this? should we feature flag the datafusion code/path, so you have to "opt-in" to compiling an image with it? we could also add an env var to actually enable the endpoint?

alexanderbianchi · 2026-04-08T20:58:03Z

+    // gRPC handler. No downstream-specific code needed here.
+    let datafusion_session_builder = if node_config
+        .is_service_enabled(QuickwitService::Searcher)
+        && quickwit_common::get_bool_from_env("QW_ENABLE_DATAFUSION_ENDPOINT", false)


@mattmkim the session is behind a env var already FYI. Open to other rec's for isolating.

ah, missed that 👍

alexanderbianchi · 2026-04-08T21:15:53Z

@@ -0,0 +1,252 @@
+// Copyright 2021-Present Datadog, Inc.


@guilload would be good to have a quickwit expert review this part

- fix: CAST unwrapping in classify_filter — reuse predicate::column_name so time-range predicates are correctly classified as Inexact and passed to scan(); previously CAST-wrapped filters were silently dropped - fix: declare parquet sort order (metric_name, timestamp_secs ASC) on FileScanConfig so DataFusion avoids redundant sort operators - fix: get_opts now respects GetOptions.range — dispatches to get_slice for Bounded/Suffix ranges instead of always downloading the full file - fix: to_object_store_error propagates file path on NotFound - fix: register_for_worker made a no-op; lazy scan-path registration is sufficient and avoids O(indexes) metastore RPCs per worker task; removes stale comment claiming a non-existent object-store cache - fix: extract is_index_not_found helper, removing duplicated downcast block from try_consume_read_rel and create_default_table_provider - fix: sort before dedup in QuickwitSchemaProvider::table_names - fix: empty searcher pool returns Ok(vec![]) for local execution fallback - fix: remove dead builder methods with_udf_batch, with_codec_applier, with_physical_optimizer_rule from DataSourceContributions - feat: add tracing spans to execute_substrait and execute_sql - feat: wire 4 GiB memory limit on DataFusionSessionBuilder in serve - refactor: extract stream_to_receiver helper in gRPC handler Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

evance-br · 2026-04-15T13:20:20Z

Hi @alexanderbianchi, Please can you give more context on this effort. Are you trying to query parquet files in DataFusion via Quickwit? How is this supposed to work for a user? How Quickwit end up with the parquet files being queried.

alexanderbianchi and others added 7 commits April 7, 2026 13:36

fix(datafusion): fix ParquetWriter::new call signature in test_utils

6b1820b

Remove extra schema argument from ParquetWriter::new; the API only accepts a ParquetWriterConfig. Remove unused ParquetSchema import. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Merge branch 'bianchi/df-3-substrait' into bianchi/df-4-serve

cfdeddd

alexanderbianchi changed the title ~~feat(datafusion): wire DataFusion gRPC service + integration tests~~ feat(datafusion): DataFusion metrics query layer Apr 7, 2026

test(datafusion): remove unimplemented REST ingest test

0b3831b

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

alexanderbianchi force-pushed the bianchi/df-4-serve branch 4 times, most recently from f57d0b1 to 07f8548 Compare April 8, 2026 20:51

mattmkim approved these changes Apr 8, 2026

View reviewed changes

alexanderbianchi commented Apr 8, 2026

View reviewed changes

alexanderbianchi force-pushed the bianchi/df-4-serve branch 5 times, most recently from c850574 to 7245f1e Compare April 9, 2026 15:55

alexanderbianchi force-pushed the bianchi/df-4-serve branch from 7245f1e to 1a5a11b Compare April 9, 2026 16:25

alexanderbianchi mentioned this pull request Apr 11, 2026

feat(datafusion): tantivy-datafusion connector for SQL over tantivy splits alexanderbianchi/quickwit#1

Draft

5 tasks

seperate dependencies with rust FF

e61b936

alexanderbianchi force-pushed the bianchi/df-4-serve branch from e97acde to e61b936 Compare April 20, 2026 17:48

alexanderbianchi added 2 commits April 20, 2026 18:03

integration testing refactor

2907fbf

object store refactor + testing

4b66f6f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(datafusion): DataFusion metrics query layer#6276

feat(datafusion): DataFusion metrics query layer#6276
alexanderbianchi wants to merge 12 commits intoquickwit-oss:mainfrom
alexanderbianchi:bianchi/df-4-serve

alexanderbianchi commented Apr 7, 2026 •

edited

Loading

Uh oh!

alexanderbianchi commented Apr 8, 2026

Uh oh!

mattmkim left a comment

Uh oh!

alexanderbianchi Apr 8, 2026

Uh oh!

mattmkim Apr 8, 2026

Uh oh!

alexanderbianchi Apr 8, 2026

Uh oh!

evance-br commented Apr 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

alexanderbianchi commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commits

Uh oh!

alexanderbianchi commented Apr 8, 2026

Uh oh!

mattmkim left a comment

Choose a reason for hiding this comment

Uh oh!

alexanderbianchi Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

mattmkim Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

alexanderbianchi Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

evance-br commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexanderbianchi commented Apr 7, 2026 •

edited

Loading

evance-br commented Apr 15, 2026 •

edited

Loading