Add streaming selection ORDER BY combine for physically sorted segments - #19120
Draft
rohityadav1993 wants to merge 2 commits into
Draft
Add streaming selection ORDER BY combine for physically sorted segments#19120rohityadav1993 wants to merge 2 commits into
rohityadav1993 wants to merge 2 commits into
Conversation
An unbounded leaf-stage ORDER BY (as injected for sorted merge join inputs) routes to MinMaxValueBasedSelectionOrderByCombineOperator, which merges every segment's rows into a single block before returning anything. At large data volumes this exceeds the leaf stage's CPU budget and ThreadAccountant raises EarlyTerminationException inside SelectionOperatorUtils.mergeWithOrdering(), surfacing to the broker as a spurious "Cancelled by sender". This adds a streaming alternative, opt-in via the `streamingSelectionOrderBy` query option: - StreamingSelectionOrderByOperator emits sorted blocks incrementally for a segment that is physically sorted on the leading ORDER BY column, reading the sorted forward index in order instead of building a priority queue. - StreamingSelectionOrderByCombineOperator performs a k-way heap merge across segment operators and emits bounded blocks (`streamingSelectionOrderByBlockSize`, default 10000) rather than one materialized result. - SelectionPlanNode and CombinePlanNode select these operators when the option is set and the sortedness precondition holds; otherwise behaviour is unchanged. Part of apache#18667.
Correctness and safety fixes on top of the streaming selection ORDER BY combine operator: - StreamingSelectionOrderByCombineOperator now checks the query deadline and termination state inside the merge loop. The merge drains its heap on the caller thread and, for single-block cursors, may never re-enter a child operator, so nothing else observed cancellation, pause, or timeout for up to (limit + offset) iterations. This restored parity with MinMaxValueBasedSelectionOrderByCombineOperator, which this operator replaces on the same query shape. - Child blocks whose data schema disagrees with the merge schema are dropped and reported as MERGE_RESPONSE errors instead of being merged blindly, mirroring SelectionOrderByResultsBlockMerger. Segments on a server can disagree on schema mid-reload, and merging rows of differing width under a single schema corrupted the result rather than failing. - Segments are released when an Error (not just an Exception) escapes the merge, so an OutOfMemoryError while accumulating output rows can no longer leave segments acquired for the lifetime of the server. In single-stage mode there is no start()/stop() backstop around this operator. - StreamingSelectionOrderByOperator reports docs and entries scanned to the shared QueryScanCostContext, as every other selection operator does. Without it, scan-based query killing was blind on this path. Also renames the query option from `streamingSelectionOrderBy` to `sortedSelectionMergeEnabled` (and `...BlockSize` to `sortedSelectionMergeBlockSize`). The option gates the classic single-stage combine path as well as the MSE leaf path, where nothing streams, so the old name misdescribed it; query option strings are effectively permanent public API. The default block size moves to a named constant. Part of apache#18667.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19120 +/- ##
============================================
+ Coverage 65.49% 65.57% +0.08%
Complexity 1423 1423
============================================
Files 3430 3434 +4
Lines 218010 218576 +566
Branches 34648 34770 +122
============================================
+ Hits 142784 143338 +554
+ Misses 63666 63655 -11
- Partials 11560 11583 +23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An unbounded leaf-stage
ORDER BY(the shape injected below a sorted merge join input, and alsoreachable directly) routes to
MinMaxValueBasedSelectionOrderByCombineOperator, which merges everysegment's rows into a single materialized block before returning anything. At large data volumes
this exceeds the leaf stage's CPU budget and
ThreadAccountantraisesEarlyTerminationExceptioninside
SelectionOperatorUtils.mergeWithOrdering(), surfacing at the broker as a spuriousCancelled by sender. This is the first bullet of Challenge 1 in #18667.This PR adds a streaming alternative for segments that are physically sorted on the leading
ORDER BYcolumn.Approach
StreamingSelectionOrderByOperator(new) emits sorted blocks incrementally for a segment thatis physically sorted on the leading
ORDER BYcolumn, walking the sorted forward index in orderinstead of filling a priority queue with the whole segment. Multi-column
ORDER BYis handled bya second pass over each equal-prefix run.
StreamingSelectionOrderByCombineOperator(new) performs a k-way heap merge across theper-segment operators and emits bounded blocks rather than one materialized result. Segments are
acquired and released incrementally, and the merge loop performs periodic termination /
deadline / resource-usage sampling, matching
BaseStreamingCombineOperator.SelectionPlanNode,CombinePlanNode,InstancePlanMakerImplV2andQueryContextselect these operators only when the option is set and the sortedness precondition holds; any
unmet precondition falls back to the existing operators.
exception rather than merged blindly, mirroring
SelectionOrderByResultsBlockMerger.Opt-in query options
sortedSelectionMergeEnabledfalsesortedSelectionMergeBlockSize10000Broker.DEFAULT_SORTED_SELECTION_MERGE_BLOCK_SIZE)No behaviour change when the option is off
Without
sortedSelectionMergeEnabled, planning and execution take the existing path unchanged — thenew operators are never constructed. No existing option, plan node, or wire format changes.
Tests
64 tests, all green:
StreamingSelectionOrderByOperatorTest(new)StreamingSelectionOrderByCombineOperatorTest(new)CombineSlowOperatorsTest(extended)QueryOptionsUtilsTest(extended)CombineSlowOperatorsTest.testStreamingSelectionOrderByCombineOperatorHonorsDeadlinepins thedeadline behaviour specifically: an already-expired deadline must yield an
ExceptionResultsBlockbefore any child operator is driven.
Known gaps
pinot-integration-tests; the broker-sidereduce,
OFFSEThandling, and the streaming response path are covered only at operator level.Happy to add one here if reviewers prefer that over a follow-up.
than per segment, which is a cost cliff on high-cardinality leading sort columns
(e.g.
ORDER BY ts, id). Correct, but worth a follow-up.Commits
This PR contains 2 commits: the feature, and a separate follow-up commit applying review fixes
(termination/deadline checks in the merge loop, schema-mismatch reporting, segment release on
Erroras well asException, scan-cost accounting, and the query-option naming above). They arekept separate so the fixes are reviewable on their own; happy to squash before merge.
Part of #18667.