[Optimize](Variant) optimize schema update performance by eldenmoon · Pull Request #45480 · apache/doris

eldenmoon · 2024-12-16T10:10:25Z

When update schema with high concurrency, updaing schemas cost is expensive.

update schema only when rows is not 0
copy_from is expensive, use copy constructor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

When update schema with high concurrency, updaing schemas cost is expensive. 1. update schema only when rows is not 0 2. copy_from is expensive, use copy constructor

Thearas · 2024-12-16T10:10:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2024-12-16T10:10:34Z

run buildall

github-actions · 2024-12-16T10:14:59Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-16T10:50:12Z

TeamCity be ut coverage result:
Function Coverage: 38.85% (10125/26064)
Line Coverage: 29.76% (85084/285909)
Region Coverage: 28.82% (43653/151486)
Branch Coverage: 25.35% (22169/87454)
Coverage Report: http://coverage.selectdb-in.cc/coverage/893082451f3bf30d28a50d4676ee25f381eb759d_893082451f3bf30d28a50d4676ee25f381eb759d/report/index.html

eldenmoon · 2024-12-16T12:37:58Z

run buildall

github-actions · 2024-12-16T12:42:16Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-16T14:07:37Z

TeamCity be ut coverage result:
Function Coverage: 38.85% (10125/26064)
Line Coverage: 29.77% (85122/285911)
Region Coverage: 28.82% (43654/151488)
Branch Coverage: 25.35% (22168/87456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9efc0374647ddd24045555eba91e2629db234735_9efc0374647ddd24045555eba91e2629db234735/report/index.html

eldenmoon · 2024-12-17T06:03:39Z

run buildall

github-actions · 2024-12-17T06:07:56Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-17T07:29:54Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26064)
Line Coverage: 29.78% (85139/285887)
Region Coverage: 28.82% (43660/151481)
Branch Coverage: 25.36% (22174/87438)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39a47e7eff29868d1f87687dddca769dfbe550e3_39a47e7eff29868d1f87687dddca769dfbe550e3/report/index.html

eldenmoon · 2024-12-17T08:32:41Z

run buildall

github-actions · 2024-12-17T08:37:20Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-17T09:02:21Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26064)
Line Coverage: 29.77% (85125/285911)
Region Coverage: 28.83% (43666/151486)
Branch Coverage: 25.37% (22182/87442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/38b5a6a8ea4e1b48f2f999f05fd733df89638eb2_38b5a6a8ea4e1b48f2f999f05fd733df89638eb2/report/index.html

csun5285 · 2024-12-20T03:48:02Z

be/src/olap/rowset_builder.cpp

-        // After adding a column v2, the schema version increases, max_version_schema needs to be updated.
-        // _tablet_schema includes k, v, and v2
-        // if v is a variant, need to add the columns decomposed from the v to the _tablet_schema.
-        if (_tablet_schema->num_variant_columns() > 0) {


may be need to revert it

eldenmoon · 2024-12-20T04:34:44Z

run buildall

eldenmoon · 2024-12-20T04:43:44Z

run buildall

csun5285

LGTM

github-actions · 2024-12-20T04:46:20Z

PR approved by anyone and no changes requested.

doris-robot · 2024-12-20T05:58:40Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10132/26063)
Line Coverage: 29.80% (85230/286006)
Region Coverage: 28.91% (43484/150397)
Branch Coverage: 25.44% (22172/87152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2ef1bea8cfef13be8c4044fe5d28ff2c3ef8a19a_2ef1bea8cfef13be8c4044fe5d28ff2c3ef8a19a/report/index.html

github-actions · 2024-12-20T08:32:14Z

PR approved by at least one committer and no changes requested.

When update schema with high concurrency, updaing schemas cost is expensive. 1. update schema only when rows is not 0 2. copy_from is expensive, use copy constructor

(#45480)

…5480 (#45730) Cherry-picked from #45480 --------- Co-authored-by: lihangyu <lihangyu@selectdb.com>

When update schema with high concurrency, updaing schemas cost is expensive. 1. update schema only when rows is not 0 2. copy_from is expensive, use copy constructor

[Optimize](Variant) optimize schema update performance

8930824

When update schema with high concurrency, updaing schemas cost is expensive. 1. update schema only when rows is not 0 2. copy_from is expensive, use copy constructor

eldenmoon added dev/2.1.x dev/3.0.x labels Dec 16, 2024

fix

39a47e7

eldenmoon force-pushed the opt-update-schema branch from 9efc037 to 39a47e7 Compare December 17, 2024 06:03

fix desc

38b5a6a

csun5285 reviewed Dec 20, 2024

View reviewed changes

eldenmoon force-pushed the opt-update-schema branch 2 times, most recently from 77eee24 to e9f59bf Compare December 20, 2024 04:33

revert

2ef1bea

eldenmoon force-pushed the opt-update-schema branch from e9f59bf to 2ef1bea Compare December 20, 2024 04:43

csun5285 approved these changes Dec 20, 2024

View reviewed changes

github-actions bot added the reviewed label Dec 20, 2024

qidaye approved these changes Dec 20, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 20, 2024

eldenmoon merged commit 62a6360 into apache:master Dec 20, 2024

eldenmoon deleted the opt-update-schema branch December 20, 2024 08:33

github-actions bot mentioned this pull request Dec 20, 2024

branch-3.0: [Optimize](Variant) optimize schema update performance #45480 #45730

Merged

github-actions bot added the dev/2.1.x-conflict label Dec 20, 2024

eldenmoon mentioned this pull request Dec 20, 2024

[Optimize](Variant) optimize schema update performance (#45480) #45731

Merged

eldenmoon added a commit that referenced this pull request Dec 21, 2024

[Optimize](Variant) optimize schema update performance (#45480) (#45731)

39c69c7

(#45480)

eldenmoon added dev/2.1.8-merged and removed dev/2.1.x-conflict dev/2.1.x labels Dec 21, 2024

eldenmoon added a commit that referenced this pull request Dec 23, 2024

branch-3.0: [Optimize](Variant) optimize schema update performance #4…

f01dd9e

…5480 (#45730) Cherry-picked from #45480 --------- Co-authored-by: lihangyu <lihangyu@selectdb.com>

eldenmoon added dev/3.0.4-merged and removed dev/3.0.x labels Dec 23, 2024

wm1581066 added the usercase Important user case type label label Feb 6, 2025

Conversation

eldenmoon commented Dec 16, 2024

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 16, 2024

Uh oh!

eldenmoon commented Dec 16, 2024

Uh oh!

github-actions bot commented Dec 16, 2024

Uh oh!

doris-robot commented Dec 16, 2024

Uh oh!

eldenmoon commented Dec 16, 2024

Uh oh!

github-actions bot commented Dec 16, 2024

Uh oh!

doris-robot commented Dec 16, 2024

Uh oh!

eldenmoon commented Dec 17, 2024

Uh oh!

github-actions bot commented Dec 17, 2024

Uh oh!

doris-robot commented Dec 17, 2024

Uh oh!

eldenmoon commented Dec 17, 2024

Uh oh!

github-actions bot commented Dec 17, 2024

Uh oh!

doris-robot commented Dec 17, 2024

Uh oh!

csun5285 Dec 20, 2024

Choose a reason for hiding this comment

Uh oh!

eldenmoon Dec 20, 2024

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Dec 20, 2024

Uh oh!

eldenmoon commented Dec 20, 2024

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

github-actions bot commented Dec 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants