Skip to content

[Improvement](shuffle) Reduce memory consumption in data stream sender#41676

Merged
Gabriel39 merged 2 commits intoapache:masterfrom
Gabriel39:opt_1010
Oct 11, 2024
Merged

[Improvement](shuffle) Reduce memory consumption in data stream sender#41676
Gabriel39 merged 2 commits intoapache:masterfrom
Gabriel39:opt_1010

Conversation

@Gabriel39
Copy link
Copy Markdown
Contributor

@Gabriel39 Gabriel39 commented Oct 10, 2024

Proposed changes

Before:
image
image

After:
image

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Copy Markdown
Contributor Author

Gabriel39 commented Oct 10, 2024

run buildall

github-actions[bot]
github-actions Bot reviewed Oct 10, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment thread be/src/vec/runtime/vdata_stream_recvr.cpp Outdated
}

void VDataStreamRecvr::add_block(Block* block, int sender_id, bool use_move) {
void VDataStreamRecvr::add_block(Block* block, int sender_id) {
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'add_block' can be made const [readability-make-member-function-const]

Suggested change
void VDataStreamRecvr::add_block(Block* block, int sender_id) {
void VDataStreamRecvr::add_block(Block* block, int sender_id) const {

be/src/vec/runtime/vdata_stream_recvr.h:88:

-     void add_block(Block* block, int sender_id);
+     void add_block(Block* block, int sender_id) const;

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 10, 2024

TeamCity be ut coverage result:
Function Coverage: 37.41% (9661/25825)
Line Coverage: 28.69% (80175/279456)
Region Coverage: 28.12% (41447/147402)
Branch Coverage: 24.72% (21107/85390)
Coverage Report: http://coverage.selectdb-in.cc/coverage/656146a35845ffad277388df9735524395cfa526_656146a35845ffad277388df9735524395cfa526/report/index.html

HappenLee
HappenLee previously approved these changes Oct 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 11, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Copy Markdown
Contributor Author

Gabriel39 commented Oct 11, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 11, 2024

TeamCity be ut coverage result:
Function Coverage: 37.40% (9660/25827)
Line Coverage: 28.67% (80136/279497)
Region Coverage: 28.11% (41445/147457)
Branch Coverage: 24.71% (21105/85420)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bd85447159141deb697f2958cef258a369b49b0e_bd85447159141deb697f2958cef258a369b49b0e/report/index.html

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 11, 2024

PR approved by at least one committer and no changes requested.

@Gabriel39 Gabriel39 merged commit af95818 into apache:master Oct 11, 2024
@Gabriel39 Gabriel39 mentioned this pull request Oct 12, 2024
Merged
Gabriel39 added a commit that referenced this pull request Oct 12, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (#41740)
cce5425 
Follow-up for #41676
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[Improvement](shuffle) Reduce memory consumption in data stream sender (
227cc8c 
apache#41676)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
4be7ee5 
)

Follow-up for apache#41676
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
58c9653 
)

Follow-up for apache#41676
dataroaring pushed a commit that referenced this pull request Oct 15, 2024
@Gabriel39
[pick](branch-3.0) pick #39982 #40576 #41555 #41592 #41676 #41740 (#4…
54b4fa3 
…1808)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024
@Gabriel39
[Improvement](shuffle) Reduce memory consumption in data stream sender (
43bb8b1 
apache#41676)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
e90a846 
)

Follow-up for apache#41676
@Gabriel39 Gabriel39 mentioned this pull request Oct 15, 2024
Merged
Gabriel39 added a commit that referenced this pull request Oct 15, 2024
@Gabriel39
[pick](branch-2.1) pick #41676 #41740 #41857 (#41904)
b185dfc 
pick #41676 #41740 #41857
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
272a8ae 
)

Follow-up for apache#41676
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@Mryange Mryange Mryange approved these changes

@HappenLee HappenLee HappenLee left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Gabriel39 @doris-robot @BiteTheDDDDt @HappenLee @Mryange