Skip to content

[fix] (compaction) fix compaction score in time series policy#40242

Merged
airborne12 merged 1 commit intoapache:masterfrom
csun5285:fix_compaction_score_calc
Sep 4, 2024
Merged

[fix] (compaction) fix compaction score in time series policy#40242
airborne12 merged 1 commit intoapache:masterfrom
csun5285:fix_compaction_score_calc

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented Sep 2, 2024 •
edited
Loading

BEFORE

The compaction score is 0 when the merge conditions are not met in the time series policy.

AFTER

The compaction score is the sum of the compaction scores of the rowsets to be merged.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 2, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Sep 2, 2024

run buildall

1 similar comment
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Sep 2, 2024

run buildall

@csun5285 csun5285 closed this Sep 2, 2024
@csun5285 csun5285 reopened this Sep 2, 2024
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Sep 2, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 2, 2024

TPC-H: Total hot run time: 37829 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3171e5165216f0c43e342686381c9aa3e0dae63, data reload: false

------ Round 1 ----------------------------------
q1	17647	4382	4291	4291
q2	2011	189	177	177
q3	11804	987	1073	987
q4	10521	712	681	681
q5	7743	2814	2777	2777
q6	224	141	133	133
q7	932	614	598	598
q8	9318	2041	2050	2041
q9	7085	6531	6518	6518
q10	6996	2183	2164	2164
q11	457	236	235	235
q12	396	225	228	225
q13	17869	3048	3045	3045
q14	283	234	233	233
q15	521	507	495	495
q16	577	511	513	511
q17	974	697	728	697
q18	7463	6868	6871	6868
q19	1390	1039	1026	1026
q20	667	335	335	335
q21	4264	2890	2761	2761
q22	1100	1051	1031	1031
Total cold run time: 110242 ms
Total hot run time: 37829 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4342	4330	4330
q2	390	276	266	266
q3	2921	2670	2622	2622
q4	1908	1611	1652	1611
q5	5623	5713	5713	5713
q6	222	134	138	134
q7	2253	1808	1848	1808
q8	3309	3373	3406	3373
q9	8894	8880	9049	8880
q10	3630	3490	3476	3476
q11	614	551	522	522
q12	855	722	714	714
q13	14999	3467	3361	3361
q14	345	315	299	299
q15	538	516	495	495
q16	609	600	575	575
q17	1839	1554	1537	1537
q18	8292	7978	7821	7821
q19	2737	1464	1538	1464
q20	2119	1908	1919	1908
q21	5617	5440	5543	5440
q22	1152	1098	1105	1098
Total cold run time: 73255 ms
Total hot run time: 57447 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 2, 2024

TPC-DS: Total hot run time: 191542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3171e5165216f0c43e342686381c9aa3e0dae63, data reload: false

query1	1266	852	871	852
query2	6287	1977	1919	1919
query3	10595	3873	3948	3873
query4	60194	25864	23221	23221
query5	5485	496	486	486
query6	416	157	173	157
query7	5788	291	286	286
query8	288	201	208	201
query9	8669	2494	2463	2463
query10	467	268	258	258
query11	18253	14994	15313	14994
query12	157	100	99	99
query13	1512	398	374	374
query14	10165	6429	7424	6429
query15	228	179	173	173
query16	7671	493	460	460
query17	1113	576	576	576
query18	2092	299	320	299
query19	290	149	144	144
query20	122	110	107	107
query21	205	105	102	102
query22	4559	4502	4305	4305
query23	34384	33850	33525	33525
query24	5968	2822	2872	2822
query25	533	370	367	367
query26	683	153	148	148
query27	1774	281	273	273
query28	3740	2046	2025	2025
query29	676	394	407	394
query30	238	154	158	154
query31	940	734	744	734
query32	83	54	52	52
query33	451	303	294	294
query34	868	470	471	470
query35	814	728	715	715
query36	1065	910	918	910
query37	142	101	95	95
query38	3995	3883	3957	3883
query39	1446	1486	1397	1397
query40	195	116	112	112
query41	45	44	43	43
query42	112	93	93	93
query43	516	450	477	450
query44	1080	734	740	734
query45	200	163	161	161
query46	1081	745	713	713
query47	1908	1812	1804	1804
query48	372	298	289	289
query49	752	410	426	410
query50	819	418	400	400
query51	7097	7063	7091	7063
query52	95	87	86	86
query53	251	175	176	175
query54	564	461	453	453
query55	74	76	76	76
query56	273	253	258	253
query57	1220	1080	1081	1080
query58	218	249	219	219
query59	2941	2695	2906	2695
query60	295	258	260	258
query61	99	93	100	93
query62	738	638	648	638
query63	213	180	185	180
query64	2732	669	686	669
query65	3226	3174	3147	3147
query66	669	327	337	327
query67	15273	15353	15297	15297
query68	2893	564	589	564
query69	412	282	278	278
query70	1197	1079	1117	1079
query71	361	279	278	278
query72	5382	4019	3939	3939
query73	747	329	325	325
query74	9118	8866	8873	8866
query75	3357	2683	2719	2683
query76	1432	1023	939	939
query77	524	309	320	309
query78	9757	9634	9022	9022
query79	1042	542	518	518
query80	684	508	510	508
query81	473	236	236	236
query82	236	146	142	142
query83	165	154	154	154
query84	257	81	83	81
query85	779	342	389	342
query86	314	289	295	289
query87	4550	4321	4214	4214
query88	3272	2342	2323	2323
query89	395	288	284	284
query90	1751	185	185	185
query91	119	95	98	95
query92	57	48	49	48
query93	1065	527	528	527
query94	585	296	287	287
query95	339	257	252	252
query96	577	263	262	262
query97	3216	3122	3105	3105
query98	219	200	197	197
query99	1501	1257	1270	1257
Total cold run time: 303456 ms
Total hot run time: 191542 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 2, 2024

ClickBench: Total hot run time: 31.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3171e5165216f0c43e342686381c9aa3e0dae63, data reload: false

query1	0.05	0.05	0.03
query2	0.08	0.05	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.08
query5	0.51	0.49	0.48
query6	1.13	0.75	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.48	0.47
query10	0.55	0.55	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	2.08	2.07	2.08
query15	0.89	0.82	0.82
query16	0.37	0.38	0.38
query17	1.05	1.04	1.05
query18	0.21	0.20	0.19
query19	1.84	1.80	1.74
query20	0.01	0.01	0.02
query21	15.40	0.67	0.67
query22	4.37	7.00	1.94
query23	18.22	1.37	1.29
query24	2.09	0.24	0.23
query25	0.16	0.08	0.08
query26	0.27	0.18	0.16
query27	0.08	0.08	0.08
query28	13.21	1.03	1.00
query29	12.65	3.28	3.29
query30	0.24	0.05	0.06
query31	2.89	0.40	0.40
query32	3.24	0.47	0.47
query33	2.94	3.03	3.00
query34	17.12	4.32	4.37
query35	4.45	4.44	4.42
query36	0.67	0.48	0.48
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.95 s
Total hot run time: 31.96 s

zzzxl1993
zzzxl1993 approved these changes Sep 2, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 2, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 4, 2024

TeamCity be ut coverage result:
Function Coverage: 36.84% (9394/25496)
Line Coverage: 28.28% (77462/273893)
Region Coverage: 27.67% (39962/144413)
Branch Coverage: 24.33% (20342/83624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3171e5165216f0c43e342686381c9aa3e0dae63_d3171e5165216f0c43e342686381c9aa3e0dae63/report/index.html

airborne12
airborne12 approved these changes Sep 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

@airborne12 airborne12 merged commit f0d936a into apache:master Sep 4, 2024
This was referenced Sep 13, 2024
Merged
Merged
yiguolei pushed a commit that referenced this pull request Sep 13, 2024
@csun5285
[fix] (compaction) fix compaction score in time series policy (#40242) (
873f70c 
#40779)

## Proposed changes

pick from master #40242

<!--Describe your changes.-->
xiaokang pushed a commit that referenced this pull request Sep 22, 2024
@csun5285
[fix] (compaction) fix compaction score in time series policy (#40242) (
99e6137 
#40789)
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@csun5285 @dataroaring
[fix] (compaction) fix compaction score in time series policy (#40242)
89b5a7a 
### BEFORE

The compaction score is 0 when the merge conditions are not met in the
time series policy.

### AFTER

The compaction score is the sum of the compaction scores of the rowsets
to be merged.
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 22, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
440543d 
…apache#40242) (apache#40779)"

This reverts commit 873f70c.
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 22, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
2b4fe14 
…apache#40242)"

This reverts commit 89b5a7a.
@csun5285 csun5285 mentioned this pull request Nov 22, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 22, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
2f70522 
…#40242)" (#44491)

revert #40242
morningman pushed a commit that referenced this pull request Nov 25, 2024
@csun5285 @yiguolei
Revert "[fix] (compaction) fix compaction score in time series policy (…
0344ed3 
…#40242) (#40779)" (#225)

This reverts commit 873f70c.
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 25, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
13047a2 
…apache#40242)" (apache#44491)

revert apache#40242
@csun5285 csun5285 mentioned this pull request Nov 25, 2024
Merged
16 tasks
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 25, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
4b487c7 
…apache#40242)" (apache#44491)

revert apache#40242
@csun5285 csun5285 mentioned this pull request Nov 25, 2024
Merged
16 tasks
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 25, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
18ca537 
…apache#40242) (apache#40779)"

This reverts commit 873f70c.
yiguolei pushed a commit that referenced this pull request Nov 25, 2024
@csun5285 @yiguolei
Revert "[fix] (compaction) fix compaction score in time series policy (…
81f1c3d 
…#40242) (#40779)" (#225)

This reverts commit 873f70c.
airborne12 pushed a commit that referenced this pull request Nov 26, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
cf2ee23 
…#40242)"  (#44527)

revert #40242
csun5285 added a commit to csun5285/doris that referenced this pull request Dec 18, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy (…
ef6962c 
…apache#40242)" (apache#44491)

revert apache#40242
airborne12 pushed a commit that referenced this pull request Dec 18, 2024
@csun5285
Revert "[fix] (compaction) fix compaction score in time series policy(#…
f167700 
…40242)" (#44525)

revert #40242
deardeng pushed a commit to deardeng/incubator-doris that referenced this pull request Jan 6, 2025
@xiaokang
Revert "[fix] (compaction) fix compaction score in time series policy (…
878cafe 
…apache#40242) (apache#40779)"

This reverts commit 873f70c.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@luwei16 luwei16 luwei16 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@csun5285 @doris-robot @airborne12 @luwei16 @zzzxl1993 @xiaokang @yiguolei @dataroaring