[fix](cloud-mow) Add enable config for procesing old version delete bitmap when doing cu compaction by hust-hhb · Pull Request #42471 · apache/doris

hust-hhb · 2024-10-25T06:45:09Z

pr #40204 support deleting old version delete bitmap when doing cu compaction, it a new function and better to add a config to enable it or not.

…itmap when doing cu compaction

doris-robot · 2024-10-25T06:45:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hust-hhb · 2024-10-25T06:46:25Z

run buildall

github-actions · 2024-10-25T06:50:02Z

clang-tidy review says "All clean, LGTM! 👍"

hust-hhb · 2024-10-25T07:35:02Z

run buildall

github-actions · 2024-10-25T07:38:50Z

clang-tidy review says "All clean, LGTM! 👍"

zhannngchen

LGTM

github-actions · 2024-10-25T07:40:20Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-25T07:40:22Z

PR approved by anyone and no changes requested.

doris-robot · 2024-10-25T09:13:16Z

TPC-H: Total hot run time: 42045 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

------ Round 1 ----------------------------------
q1	17572	7519	7324	7324
q2	2033	284	276	276
q3	12370	1081	1179	1081
q4	10556	900	926	900
q5	7774	3162	3139	3139
q6	243	151	151	151
q7	1038	630	621	621
q8	9379	2027	2033	2027
q9	6565	6464	6442	6442
q10	7084	2459	2447	2447
q11	446	256	254	254
q12	410	221	219	219
q13	17787	3018	3027	3018
q14	238	223	236	223
q15	562	514	512	512
q16	646	593	585	585
q17	985	548	569	548
q18	7519	6726	6800	6726
q19	1340	1061	1072	1061
q20	493	183	183	183
q21	4099	3312	3314	3312
q22	1130	1022	996	996
Total cold run time: 110269 ms
Total hot run time: 42045 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7289	7553	7289
q2	347	246	237	237
q3	3098	3050	3070	3050
q4	2175	1908	1825	1825
q5	5791	5771	5812	5771
q6	229	141	143	141
q7	2264	1824	1865	1824
q8	3418	3460	3504	3460
q9	9017	8896	8960	8896
q10	3605	3599	3600	3599
q11	586	491	503	491
q12	831	623	608	608
q13	10490	3195	3174	3174
q14	318	282	277	277
q15	579	523	516	516
q16	683	625	643	625
q17	1856	1629	1649	1629
q18	8284	7699	7584	7584
q19	1715	1543	1608	1543
q20	2116	1871	1905	1871
q21	5495	5487	5546	5487
q22	1167	1099	1050	1050
Total cold run time: 71408 ms
Total hot run time: 60947 ms

doris-robot · 2024-10-25T09:23:31Z

TeamCity be ut coverage result:
Function Coverage: 37.45% (9712/25932)
Line Coverage: 28.71% (80534/280553)
Region Coverage: 28.14% (41651/148019)
Branch Coverage: 24.69% (21160/85692)
Coverage Report: http://coverage.selectdb-in.cc/coverage/640607ee3dd69714932ade61aa4fe8537ab2c9a9_640607ee3dd69714932ade61aa4fe8537ab2c9a9/report/index.html

doris-robot · 2024-10-25T09:25:17Z

TPC-DS: Total hot run time: 192444 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

query1	845	411	410	410
query2	6258	2102	2092	2092
query3	8673	196	202	196
query4	35027	23806	23635	23635
query5	4729	458	440	440
query6	278	167	173	167
query7	4198	298	306	298
query8	288	232	233	232
query9	9570	2679	2682	2679
query10	485	261	248	248
query11	18085	15293	15238	15238
query12	155	112	103	103
query13	1600	440	430	430
query14	9633	7021	7540	7021
query15	252	168	184	168
query16	7683	497	516	497
query17	1520	602	610	602
query18	2065	315	308	308
query19	379	163	160	160
query20	123	120	116	116
query21	208	106	112	106
query22	4892	4719	4563	4563
query23	34689	34092	34876	34092
query24	10714	2782	2816	2782
query25	589	407	405	405
query26	723	165	172	165
query27	1989	281	295	281
query28	6489	2471	2449	2449
query29	733	440	417	417
query30	257	158	169	158
query31	1034	793	806	793
query32	98	53	56	53
query33	660	284	277	277
query34	905	499	515	499
query35	986	895	888	888
query36	1106	954	948	948
query37	120	72	77	72
query38	4355	4307	4271	4271
query39	1456	1444	1455	1444
query40	202	103	106	103
query41	51	48	50	48
query42	113	100	101	100
query43	538	515	493	493
query44	1189	825	814	814
query45	184	166	168	166
query46	1111	706	738	706
query47	1973	1869	1869	1869
query48	426	330	321	321
query49	866	409	422	409
query50	807	396	391	391
query51	7038	7040	6834	6834
query52	106	89	97	89
query53	267	184	181	181
query54	1236	420	415	415
query55	79	79	78	78
query56	262	271	241	241
query57	1307	1145	1148	1145
query58	212	197	202	197
query59	3386	3141	3003	3003
query60	276	252	242	242
query61	106	102	106	102
query62	833	671	646	646
query63	215	194	187	187
query64	3730	644	615	615
query65	3285	3247	3196	3196
query66	716	307	299	299
query67	16242	15652	15827	15652
query68	4451	577	570	570
query69	424	254	288	254
query70	1233	1155	1152	1152
query71	316	257	270	257
query72	6248	4038	4011	4011
query73	778	361	361	361
query74	10033	9003	9063	9003
query75	3415	2668	2672	2668
query76	2760	1003	976	976
query77	401	279	275	275
query78	10586	9576	9538	9538
query79	1736	589	598	589
query80	1078	429	435	429
query81	559	237	246	237
query82	664	121	114	114
query83	233	135	138	135
query84	237	77	68	68
query85	1242	295	283	283
query86	365	317	304	304
query87	4861	4595	4650	4595
query88	3390	2236	2187	2187
query89	410	287	289	287
query90	1940	191	194	191
query91	133	102	106	102
query92	62	49	46	46
query93	1861	556	545	545
query94	737	301	310	301
query95	349	241	244	241
query96	636	286	286	286
query97	2923	2702	2706	2702
query98	219	199	198	198
query99	1788	1336	1294	1294
Total cold run time: 296673 ms
Total hot run time: 192444 ms

doris-robot · 2024-10-25T09:30:49Z

ClickBench: Total hot run time: 32.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.41	0.40	0.40
query6	1.13	0.66	0.64
query7	0.02	0.01	0.01
query8	0.06	0.03	0.03
query9	0.56	0.50	0.49
query10	0.57	0.53	0.55
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.72	2.71	2.80
query15	0.89	0.83	0.85
query16	0.38	0.38	0.39
query17	1.05	1.06	1.03
query18	0.21	0.20	0.21
query19	1.84	1.76	1.88
query20	0.01	0.01	0.01
query21	15.35	0.61	0.61
query22	3.21	2.31	1.73
query23	17.17	0.91	0.82
query24	3.03	1.38	0.88
query25	0.23	0.21	0.09
query26	0.41	0.14	0.14
query27	0.04	0.04	0.03
query28	10.45	1.10	1.08
query29	12.65	3.35	3.31
query30	0.24	0.05	0.06
query31	2.87	0.39	0.38
query32	3.26	0.46	0.46
query33	2.99	3.02	3.07
query34	17.26	4.53	4.47
query35	4.53	4.53	4.52
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.57 s
Total hot run time: 32.42 s

dataroaring

LGTM

zhannngchen · 2024-10-30T07:43:13Z

run buildall

doris-robot · 2024-10-30T09:27:02Z

TPC-H: Total hot run time: 41359 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

------ Round 1 ----------------------------------
q1	17727	7598	7339	7339
q2	2071	175	160	160
q3	10670	1114	1179	1114
q4	10367	911	858	858
q5	7757	3090	3079	3079
q6	233	147	152	147
q7	1034	604	606	604
q8	9349	1945	2031	1945
q9	6595	6459	6515	6459
q10	7081	2440	2429	2429
q11	454	242	246	242
q12	409	211	207	207
q13	17797	3008	3011	3008
q14	255	216	224	216
q15	580	512	508	508
q16	655	594	591	591
q17	979	555	530	530
q18	7408	6680	6740	6680
q19	1349	1017	1055	1017
q20	459	183	180	180
q21	3997	3258	3062	3062
q22	1130	1032	984	984
Total cold run time: 108356 ms
Total hot run time: 41359 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7382	7254	7302	7254
q2	325	235	233	233
q3	3014	2987	2993	2987
q4	2107	1848	1799	1799
q5	5731	5751	5808	5751
q6	229	145	141	141
q7	2237	1825	1809	1809
q8	3381	3558	3449	3449
q9	9067	9082	8989	8989
q10	3597	3609	3562	3562
q11	598	494	493	493
q12	837	639	623	623
q13	8689	3178	3201	3178
q14	311	279	289	279
q15	579	525	516	516
q16	698	650	653	650
q17	1857	1635	1654	1635
q18	8301	7847	7683	7683
q19	1750	1598	1591	1591
q20	2117	1845	1923	1845
q21	5654	5389	5489	5389
q22	1155	1066	1066	1066
Total cold run time: 69616 ms
Total hot run time: 60922 ms

doris-robot · 2024-10-30T09:39:13Z

TPC-DS: Total hot run time: 196763 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

query1	1260	962	963	962
query2	6226	2090	2059	2059
query3	11479	4763	4899	4763
query4	33860	23609	23590	23590
query5	5074	465	429	429
query6	269	178	158	158
query7	3984	291	296	291
query8	279	224	223	223
query9	9692	2613	2621	2613
query10	506	273	242	242
query11	18259	15448	15123	15123
query12	161	102	103	102
query13	1579	405	421	405
query14	9577	7185	7379	7185
query15	256	180	179	179
query16	7789	511	505	505
query17	1420	601	584	584
query18	2111	308	306	306
query19	385	157	152	152
query20	121	111	118	111
query21	208	105	111	105
query22	5183	4777	4496	4496
query23	34527	34049	34068	34049
query24	11139	2726	2793	2726
query25	624	401	408	401
query26	754	157	152	152
query27	2063	284	291	284
query28	6822	2407	2406	2406
query29	770	415	420	415
query30	260	164	157	157
query31	1037	799	821	799
query32	90	55	59	55
query33	767	266	274	266
query34	939	500	518	500
query35	1008	857	894	857
query36	1108	945	947	945
query37	129	78	71	71
query38	4352	4274	4295	4274
query39	1432	1415	1437	1415
query40	203	100	100	100
query41	48	44	46	44
query42	110	94	97	94
query43	534	499	489	489
query44	1233	791	806	791
query45	178	164	162	162
query46	1140	712	697	697
query47	1981	1899	1899	1899
query48	401	310	320	310
query49	899	406	392	392
query50	800	385	388	385
query51	7062	7151	6904	6904
query52	94	85	87	85
query53	254	179	181	179
query54	1235	413	406	406
query55	76	75	76	75
query56	268	254	246	246
query57	1307	1181	1154	1154
query58	215	194	199	194
query59	3222	2995	3145	2995
query60	268	241	243	241
query61	102	101	96	96
query62	863	682	681	681
query63	210	185	187	185
query64	4022	637	603	603
query65	3273	3165	3183	3165
query66	783	303	296	296
query67	16194	15875	15735	15735
query68	4534	557	529	529
query69	440	258	269	258
query70	1215	1159	1157	1157
query71	356	248	241	241
query72	6258	3899	3967	3899
query73	788	355	352	352
query74	10303	8959	9021	8959
query75	3402	2616	2712	2616
query76	2469	1100	1049	1049
query77	370	269	267	267
query78	10620	9534	9564	9534
query79	1103	588	572	572
query80	776	406	416	406
query81	539	239	238	238
query82	1350	116	112	112
query83	223	136	135	135
query84	237	69	70	69
query85	915	291	273	273
query86	331	294	300	294
query87	4827	4573	4735	4573
query88	2940	2200	2152	2152
query89	406	287	276	276
query90	1781	181	181	181
query91	133	97	96	96
query92	57	49	50	49
query93	1067	535	526	526
query94	725	284	294	284
query95	345	237	241	237
query96	616	281	281	281
query97	2935	2720	2678	2678
query98	208	203	198	198
query99	1692	1339	1314	1314
Total cold run time: 298435 ms
Total hot run time: 196763 ms

doris-robot · 2024-10-30T09:39:16Z

TeamCity be ut coverage result:
Function Coverage: 37.94% (9845/25949)
Line Coverage: 29.22% (81998/280636)
Region Coverage: 28.57% (42296/148048)
Branch Coverage: 25.10% (21498/85664)
Coverage Report: http://coverage.selectdb-in.cc/coverage/640607ee3dd69714932ade61aa4fe8537ab2c9a9_640607ee3dd69714932ade61aa4fe8537ab2c9a9/report/index.html

doris-robot · 2024-10-30T09:44:43Z

ClickBench: Total hot run time: 32.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 640607ee3dd69714932ade61aa4fe8537ab2c9a9, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.42	0.38	0.42
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.56	0.50	0.50
query10	0.55	0.57	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	2.74	2.73	2.74
query15	0.90	0.82	0.83
query16	0.38	0.40	0.39
query17	1.04	1.00	1.05
query18	0.20	0.20	0.21
query19	1.97	1.85	2.02
query20	0.02	0.01	0.01
query21	15.37	0.59	0.58
query22	2.29	2.64	1.95
query23	16.89	0.88	0.99
query24	2.97	1.00	1.22
query25	0.17	0.20	0.08
query26	0.43	0.15	0.14
query27	0.04	0.04	0.05
query28	10.65	1.09	1.07
query29	12.60	3.31	3.35
query30	0.24	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.46	0.46
query33	3.00	3.04	3.04
query34	16.89	4.38	4.42
query35	4.48	4.46	4.45
query36	0.69	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 32.78 s

…itmap when doing cu compaction (apache#42471) pr apache#40204 support deleting old version delete bitmap when doing cu compaction, it a new function and better to add a config to enable it or not.

…itmap when doing cu compaction (#43615) pr #40204 support deleting old version delete bitmap when doing cu compaction, it a new function and better to add a config to enable it or not. pick pr:#42471

[fix](cloud-mow) Add enable config for procesing old version delete b…

84adc8f

…itmap when doing cu compaction

zhannngchen reviewed Oct 25, 2024

View reviewed changes

Comment thread be/src/common/config.h Outdated

edit

640607e

zhannngchen approved these changes Oct 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2024

github-actions bot added the reviewed label Oct 25, 2024

dataroaring approved these changes Oct 29, 2024

View reviewed changes

dataroaring merged commit 9a907b0 into apache:master Oct 31, 2024

hust-hhb mentioned this pull request Nov 11, 2024

[fix](cloud-mow) Add enable config for procesing old version delete bitmap when doing cu compaction #43615

Merged

Conversation

hust-hhb commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Oct 25, 2024

Uh oh!

hust-hhb commented Oct 25, 2024

Uh oh!

github-actions bot commented Oct 25, 2024

Uh oh!

Uh oh!

hust-hhb commented Oct 25, 2024

Uh oh!

github-actions bot commented Oct 25, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 25, 2024

Uh oh!

github-actions bot commented Oct 25, 2024

Uh oh!

doris-robot commented Oct 25, 2024

Uh oh!

doris-robot commented Oct 25, 2024

Uh oh!

doris-robot commented Oct 25, 2024

Uh oh!

doris-robot commented Oct 25, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

zhannngchen commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hust-hhb commented Oct 25, 2024 •

edited

Loading