[opt](catalog) merge scan range to avoid too many splits (#38311)#38964
[opt](catalog) merge scan range to avoid too many splits (#38311)#38964morningman merged 2 commits intoapache:branch-2.1from
Conversation
PR apache#34032 introduce a new method to get splits batch by batch, but it removed a logic that BE will merge scan ranges to avoid too many scan ranges being scheduled. This PR mainly changes: 1. Add scan range merging logic back. 2. Change the default file split size from 8MB to 64MB, to avoid too many small split.
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| LOG(INFO) << "Merge " << scan_ranges.size() << " scan ranges to " << merged_ranges.size(); | ||
| } | ||
|
|
||
| protected: |
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
| protected: |
Additional context
be/src/vec/exec/scan/split_source_connector.h:46: previously declared here
protected:
^
|
run buildall |
|
TeamCity be ut coverage result: |
2 participants
bp #38311