Skip to content

[fix](delete-sign) Disable deletion with delete sign when doing cumulative compaction#37950

Merged
dataroaring merged 5 commits intoapache:masterfrom
Hastyshell:fix-delete-sign-with-cumu-delete
Jul 19, 2024
Merged

[fix](delete-sign) Disable deletion with delete sign when doing cumulative compaction#37950
dataroaring merged 5 commits intoapache:masterfrom
Hastyshell:fix-delete-sign-with-cumu-delete

Conversation

@Hastyshell
Copy link
Copy Markdown
Collaborator

@Hastyshell Hastyshell commented Jul 16, 2024

Proposed changes

Delete sign could not be applied when delete on cumu compaction is enabled, bucause it is meant for delete with predicates. If delete design is applied on cumu compaction, it will lose effect when doing base compaction.

So disable delete sign when doing cumulative compaction.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 16, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 16, 2024

TPC-H: Total hot run time: 39736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14a5c164190646480e9ce72bbf9e32f5a0fdfba2, data reload: false

------ Round 1 ----------------------------------
q1	17617	4265	4291	4265
q2	2011	188	194	188
q3	10450	1148	1137	1137
q4	10170	740	825	740
q5	7574	2700	2665	2665
q6	217	133	135	133
q7	951	598	596	596
q8	9217	2075	2040	2040
q9	8789	6529	6544	6529
q10	8779	3758	3793	3758
q11	457	227	235	227
q12	462	230	225	225
q13	18952	3010	2999	2999
q14	280	239	246	239
q15	541	499	490	490
q16	514	381	376	376
q17	971	711	665	665
q18	8040	7476	7328	7328
q19	7118	1383	1360	1360
q20	691	327	329	327
q21	4962	3173	3165	3165
q22	350	289	284	284
Total cold run time: 119113 ms
Total hot run time: 39736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4410	4234	4249	4234
q2	381	270	269	269
q3	2993	2889	2934	2889
q4	2118	1686	1713	1686
q5	5672	5526	5466	5466
q6	238	135	134	134
q7	2236	1891	1769	1769
q8	3257	3408	3401	3401
q9	8761	8736	8850	8736
q10	4042	3874	3797	3797
q11	604	515	494	494
q12	844	661	610	610
q13	17295	3225	3207	3207
q14	317	291	288	288
q15	529	486	509	486
q16	497	453	478	453
q17	1825	1531	1473	1473
q18	8108	8018	7736	7736
q19	1762	1588	1546	1546
q20	2110	1878	1855	1855
q21	9525	4814	4563	4563
q22	615	535	535	535
Total cold run time: 78139 ms
Total hot run time: 55627 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 16, 2024

TPC-DS: Total hot run time: 172868 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14a5c164190646480e9ce72bbf9e32f5a0fdfba2, data reload: false

query1	904	366	364	364
query2	6445	1960	1846	1846
query3	6637	210	220	210
query4	24600	17405	17226	17226
query5	3680	484	482	482
query6	270	190	176	176
query7	4595	291	283	283
query8	245	190	191	190
query9	8539	2403	2390	2390
query10	426	293	268	268
query11	10860	10156	10108	10108
query12	115	93	83	83
query13	1631	362	353	353
query14	9416	7208	7533	7208
query15	216	162	166	162
query16	7713	317	310	310
query17	1801	547	519	519
query18	1966	268	276	268
query19	189	145	147	145
query20	85	84	79	79
query21	204	125	127	125
query22	4432	4186	4080	4080
query23	34378	33742	33484	33484
query24	7892	2870	2895	2870
query25	590	392	408	392
query26	716	157	153	153
query27	2244	277	282	277
query28	6166	2059	2078	2059
query29	938	665	614	614
query30	253	153	156	153
query31	956	786	758	758
query32	96	52	56	52
query33	610	328	292	292
query34	871	498	518	498
query35	666	581	592	581
query36	1084	969	965	965
query37	148	86	83	83
query38	2990	2874	2824	2824
query39	908	858	853	853
query40	237	121	122	121
query41	47	50	43	43
query42	114	98	99	98
query43	488	493	464	464
query44	1089	715	719	715
query45	194	163	162	162
query46	1080	736	757	736
query47	1855	1749	1782	1749
query48	365	298	288	288
query49	836	409	417	409
query50	784	381	388	381
query51	6899	6837	6798	6798
query52	104	98	92	92
query53	349	281	287	281
query54	653	446	448	446
query55	75	73	73	73
query56	277	257	257	257
query57	1146	1066	1042	1042
query58	252	235	257	235
query59	2743	2580	2739	2580
query60	297	271	311	271
query61	93	93	94	93
query62	764	654	649	649
query63	311	283	286	283
query64	9108	2202	1635	1635
query65	3169	3086	3120	3086
query66	744	333	329	329
query67	15512	14982	14973	14973
query68	4461	531	540	531
query69	504	323	315	315
query70	1170	1117	1108	1108
query71	360	271	272	271
query72	7262	5348	5474	5348
query73	731	326	324	324
query74	6209	5664	5627	5627
query75	3385	2731	2639	2639
query76	2097	930	912	912
query77	423	301	297	297
query78	11349	9337	9706	9337
query79	2520	510	523	510
query80	2007	480	480	480
query81	581	218	216	216
query82	600	134	131	131
query83	284	168	167	167
query84	277	85	84	84
query85	703	306	350	306
query86	476	293	288	288
query87	3305	3049	3132	3049
query88	3847	2382	2384	2382
query89	470	380	373	373
query90	1802	187	189	187
query91	125	100	101	100
query92	56	48	51	48
query93	2892	503	493	493
query94	1109	216	212	212
query95	393	312	303	303
query96	594	271	270	270
query97	3202	3027	3038	3027
query98	216	196	201	196
query99	1700	1280	1265	1265
Total cold run time: 271170 ms
Total hot run time: 172868 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 16, 2024

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14a5c164190646480e9ce72bbf9e32f5a0fdfba2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.08	0.07
query5	0.51	0.48	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.48	0.48
query10	0.55	0.53	0.52
query11	0.14	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.58	0.58
query14	0.76	0.79	0.75
query15	0.85	0.81	0.82
query16	0.34	0.35	0.36
query17	0.93	0.96	0.97
query18	0.22	0.23	0.21
query19	1.83	1.67	1.65
query20	0.02	0.01	0.01
query21	15.39	0.74	0.65
query22	4.66	6.92	1.89
query23	18.37	1.41	1.26
query24	2.14	0.23	0.24
query25	0.14	0.08	0.09
query26	0.29	0.21	0.21
query27	0.45	0.24	0.23
query28	13.25	1.01	1.01
query29	12.82	3.32	3.27
query30	0.26	0.06	0.06
query31	2.91	0.38	0.38
query32	3.28	0.48	0.47
query33	2.86	2.90	2.89
query34	17.18	4.30	4.32
query35	4.40	4.47	4.40
query36	0.65	0.49	0.48
query37	0.18	0.15	0.17
query38	0.17	0.14	0.16
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.11	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.47 s
Total hot run time: 30.38 s

@Hastyshell Hastyshell force-pushed the fix-delete-sign-with-cumu-delete branch from fad7e07 to fec9167 Compare July 17, 2024 03:05
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-H: Total hot run time: 40038 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fec91674e701fd74f077cb8e68a5158842305e3e, data reload: false

------ Round 1 ----------------------------------
q1	18034	4478	4429	4429
q2	2466	193	194	193
q3	11518	1171	1121	1121
q4	10784	719	722	719
q5	7585	2823	2677	2677
q6	222	140	140	140
q7	963	615	602	602
q8	9317	2071	2096	2071
q9	8749	6532	6511	6511
q10	8702	3778	3748	3748
q11	439	238	253	238
q12	395	227	228	227
q13	18651	2978	2985	2978
q14	273	231	239	231
q15	531	483	491	483
q16	483	393	376	376
q17	968	666	679	666
q18	8002	7573	7464	7464
q19	5549	1421	1414	1414
q20	704	332	330	330
q21	5479	3138	3213	3138
q22	340	286	282	282
Total cold run time: 120154 ms
Total hot run time: 40038 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4220	4378	4220
q2	370	271	258	258
q3	3051	2775	2733	2733
q4	1881	1579	1597	1579
q5	5307	5293	5315	5293
q6	223	128	131	128
q7	2125	1748	1712	1712
q8	3188	3326	3308	3308
q9	8399	8324	8437	8324
q10	3900	3705	3662	3662
q11	575	483	500	483
q12	771	654	613	613
q13	17464	2966	2974	2966
q14	321	264	279	264
q15	524	485	473	473
q16	456	410	415	410
q17	1760	1467	1462	1462
q18	7682	7562	7499	7499
q19	1693	1549	1548	1548
q20	2015	1777	1807	1777
q21	6678	4726	4706	4706
q22	582	489	510	489
Total cold run time: 73329 ms
Total hot run time: 53907 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-DS: Total hot run time: 173252 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fec91674e701fd74f077cb8e68a5158842305e3e, data reload: false

query1	904	379	365	365
query2	6447	1880	1772	1772
query3	6649	221	223	221
query4	28435	17504	17336	17336
query5	4243	514	494	494
query6	269	181	190	181
query7	4584	294	289	289
query8	239	189	184	184
query9	8611	2398	2377	2377
query10	432	287	284	284
query11	11691	10124	10109	10109
query12	143	82	81	81
query13	1631	368	363	363
query14	10080	7735	7807	7735
query15	234	162	172	162
query16	7827	330	312	312
query17	1801	585	556	556
query18	1926	292	288	288
query19	209	155	167	155
query20	93	86	87	86
query21	211	130	127	127
query22	4432	4142	4286	4142
query23	33793	33307	33127	33127
query24	12182	2945	2859	2859
query25	655	366	370	366
query26	1814	146	148	146
query27	2896	271	278	271
query28	7650	1991	1995	1991
query29	1172	628	613	613
query30	273	150	147	147
query31	931	749	761	749
query32	97	51	55	51
query33	774	293	295	293
query34	997	484	492	484
query35	691	595	572	572
query36	1094	924	917	917
query37	280	81	75	75
query38	2862	2771	2763	2763
query39	867	794	816	794
query40	282	122	120	120
query41	71	45	46	45
query42	118	100	103	100
query43	515	462	460	460
query44	1184	736	725	725
query45	187	159	161	159
query46	1092	746	747	746
query47	1862	1750	1769	1750
query48	369	288	294	288
query49	1178	420	413	413
query50	794	399	398	398
query51	6837	6830	6819	6819
query52	99	93	91	91
query53	356	295	292	292
query54	924	448	453	448
query55	74	75	74	74
query56	284	260	274	260
query57	1149	1082	1065	1065
query58	256	244	260	244
query59	2899	2710	2534	2534
query60	299	280	300	280
query61	113	96	106	96
query62	828	637	657	637
query63	328	291	284	284
query64	10444	2227	1705	1705
query65	3205	3097	3113	3097
query66	1375	337	336	336
query67	15621	15335	14943	14943
query68	5123	552	553	552
query69	610	420	343	343
query70	1112	1149	1135	1135
query71	432	285	276	276
query72	7059	5705	5710	5705
query73	801	330	332	330
query74	6242	5724	5591	5591
query75	3531	2727	2652	2652
query76	3646	925	926	925
query77	653	306	325	306
query78	9733	10143	9004	9004
query79	3023	518	526	518
query80	2053	463	487	463
query81	571	222	221	221
query82	1245	146	140	140
query83	328	173	175	173
query84	276	90	91	90
query85	1670	323	306	306
query86	478	332	311	311
query87	3231	3093	3103	3093
query88	4164	2423	2373	2373
query89	477	397	387	387
query90	1829	194	195	194
query91	128	103	99	99
query92	58	52	51	51
query93	2591	508	498	498
query94	1264	214	208	208
query95	404	326	317	317
query96	603	281	275	275
query97	3152	3040	3024	3024
query98	215	205	206	205
query99	1637	1241	1260	1241
Total cold run time: 291201 ms
Total hot run time: 173252 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

ClickBench: Total hot run time: 30.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fec91674e701fd74f077cb8e68a5158842305e3e, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.53	0.50	0.48
query6	1.15	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.56	0.51	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.80	0.77
query15	0.85	0.80	0.81
query16	0.36	0.38	0.34
query17	0.99	0.95	0.99
query18	0.22	0.21	0.22
query19	1.80	1.72	1.70
query20	0.02	0.01	0.01
query21	15.37	0.77	0.65
query22	4.12	7.44	1.86
query23	18.29	1.36	1.24
query24	2.12	0.25	0.22
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.22	1.01	1.00
query29	12.65	3.26	3.23
query30	0.25	0.05	0.06
query31	2.89	0.42	0.39
query32	3.24	0.47	0.46
query33	2.93	2.85	2.96
query34	17.13	4.31	4.35
query35	4.47	4.44	4.43
query36	0.66	0.46	0.49
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.12
query41	0.09	0.06	0.06
query42	0.06	0.05	0.04
query43	0.05	0.04	0.05
Total cold run time: 109.77 s
Total hot run time: 30.37 s

@Hastyshell Hastyshell force-pushed the fix-delete-sign-with-cumu-delete branch from fec9167 to e9725f1 Compare July 17, 2024 06:30
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Hastyshell
Copy link
Copy Markdown
Collaborator Author

Hastyshell commented Jul 17, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-H: Total hot run time: 39459 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9725f15ee90f5859a39c2d993b797f580d353ce, data reload: false

------ Round 1 ----------------------------------
q1	17611	4389	4263	4263
q2	2013	187	190	187
q3	10478	1212	1070	1070
q4	10196	766	767	766
q5	7524	2734	2664	2664
q6	221	135	135	135
q7	941	596	597	596
q8	9224	2052	2073	2052
q9	8730	6541	6520	6520
q10	8809	3793	3763	3763
q11	461	233	228	228
q12	450	223	223	223
q13	17759	2951	2980	2951
q14	274	231	233	231
q15	526	492	477	477
q16	498	378	378	378
q17	958	630	696	630
q18	8088	7415	7270	7270
q19	5295	1446	1230	1230
q20	691	312	324	312
q21	5066	3235	3260	3235
q22	338	278	280	278
Total cold run time: 116151 ms
Total hot run time: 39459 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4199	4239	4199
q2	366	268	261	261
q3	3035	2813	2937	2813
q4	1947	1723	1681	1681
q5	5673	5552	5501	5501
q6	228	133	134	133
q7	2154	1806	1857	1806
q8	3341	3413	3350	3350
q9	8797	8728	8785	8728
q10	4125	4011	3714	3714
q11	576	478	516	478
q12	812	665	622	622
q13	17189	3168	3185	3168
q14	309	292	291	291
q15	517	498	486	486
q16	516	435	434	434
q17	1787	1554	1509	1509
q18	8043	8154	7777	7777
q19	1769	1596	1625	1596
q20	2140	1871	1842	1842
q21	5101	4950	4886	4886
q22	610	501	510	501
Total cold run time: 73400 ms
Total hot run time: 55776 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-DS: Total hot run time: 173359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9725f15ee90f5859a39c2d993b797f580d353ce, data reload: false

query1	900	366	377	366
query2	6461	1915	1832	1832
query3	6639	211	239	211
query4	24944	17409	17556	17409
query5	3676	477	480	477
query6	262	186	168	168
query7	4584	290	293	290
query8	248	184	211	184
query9	8485	2400	2372	2372
query10	431	274	263	263
query11	12687	10217	10129	10129
query12	110	82	80	80
query13	1638	361	352	352
query14	10129	7643	7264	7264
query15	211	160	164	160
query16	7505	313	307	307
query17	1369	545	555	545
query18	1745	279	267	267
query19	201	146	144	144
query20	89	76	80	76
query21	205	134	123	123
query22	4454	4187	4363	4187
query23	33747	33609	33600	33600
query24	10828	2929	2900	2900
query25	581	388	383	383
query26	740	150	163	150
query27	2234	272	279	272
query28	5913	2066	2058	2058
query29	924	641	603	603
query30	254	153	156	153
query31	974	778	753	753
query32	98	54	58	54
query33	691	341	323	323
query34	883	484	496	484
query35	715	579	584	579
query36	1100	966	966	966
query37	145	87	122	87
query38	2972	2882	2871	2871
query39	908	837	859	837
query40	200	117	120	117
query41	48	43	44	43
query42	116	102	102	102
query43	507	451	460	451
query44	1054	724	742	724
query45	192	161	163	161
query46	1094	747	734	734
query47	1888	1754	1778	1754
query48	373	293	284	284
query49	823	396	404	396
query50	775	381	374	374
query51	6855	6857	6798	6798
query52	107	95	87	87
query53	362	278	279	278
query54	906	450	435	435
query55	78	70	72	70
query56	284	260	281	260
query57	1135	1069	1069	1069
query58	281	232	259	232
query59	2766	2611	2564	2564
query60	308	266	271	266
query61	95	91	94	91
query62	783	620	649	620
query63	334	282	285	282
query64	9111	2158	1646	1646
query65	3165	3097	3079	3079
query66	745	324	328	324
query67	15471	15092	15080	15080
query68	4759	593	553	553
query69	665	400	341	341
query70	1206	1129	1126	1126
query71	406	275	272	272
query72	7767	5516	6268	5516
query73	762	325	326	325
query74	6273	5606	5669	5606
query75	3387	2697	2725	2697
query76	2875	909	941	909
query77	715	292	311	292
query78	9637	9060	13033	9060
query79	3080	514	527	514
query80	1944	483	460	460
query81	583	217	228	217
query82	426	130	132	130
query83	326	170	165	165
query84	267	89	82	82
query85	1418	293	288	288
query86	476	312	304	304
query87	3284	3153	3131	3131
query88	3308	2377	2371	2371
query89	467	368	375	368
query90	1760	188	186	186
query91	128	96	99	96
query92	61	47	48	47
query93	1071	500	501	500
query94	1194	208	204	204
query95	397	307	309	307
query96	583	274	267	267
query97	3283	2983	3007	2983
query98	292	198	195	195
query99	1564	1253	1247	1247
Total cold run time: 274900 ms
Total hot run time: 173359 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

ClickBench: Total hot run time: 31.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9725f15ee90f5859a39c2d993b797f580d353ce, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.16	0.12	0.13
query13	0.60	0.58	0.58
query14	0.78	0.76	0.78
query15	0.85	0.81	0.82
query16	0.36	0.35	0.37
query17	0.98	0.95	0.97
query18	0.22	0.22	0.22
query19	1.81	1.68	1.81
query20	0.01	0.01	0.01
query21	15.39	0.73	0.65
query22	3.80	6.25	2.84
query23	18.32	1.40	1.27
query24	2.18	0.22	0.22
query25	0.15	0.09	0.08
query26	0.29	0.21	0.20
query27	0.46	0.22	0.23
query28	13.29	1.01	1.01
query29	12.63	3.29	3.32
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.28	0.48	0.47
query33	2.83	2.99	2.92
query34	17.10	4.31	4.34
query35	4.45	4.39	4.39
query36	0.65	0.46	0.49
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.46 s
Total hot run time: 31.46 s

luwei16
luwei16 reviewed Jul 17, 2024
View reviewed changes
Comment thread be/src/vec/olap/block_reader.cpp Outdated
@Hastyshell Hastyshell force-pushed the fix-delete-sign-with-cumu-delete branch 2 times, most recently from 8f1e859 to 7fa67a6 Compare July 17, 2024 12:43
@Hastyshell
Copy link
Copy Markdown
Collaborator Author

Hastyshell commented Jul 17, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Hastyshell Hastyshell requested a review from luwei16 July 17, 2024 12:46
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-H: Total hot run time: 40086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7fa67a690bc24136b230258c0855f4319a338bc7, data reload: false

------ Round 1 ----------------------------------
q1	17877	5028	4393	4393
q2	2657	194	195	194
q3	11216	1186	1097	1097
q4	10451	780	888	780
q5	7894	2732	2809	2732
q6	230	138	140	138
q7	974	623	623	623
q8	9346	2096	2129	2096
q9	8799	6534	6551	6534
q10	8820	3742	3780	3742
q11	472	236	245	236
q12	400	224	223	223
q13	17761	2989	2979	2979
q14	279	241	233	233
q15	528	472	481	472
q16	492	389	385	385
q17	957	654	727	654
q18	8054	7362	7461	7362
q19	7552	1439	1363	1363
q20	649	333	324	324
q21	4968	3242	3308	3242
q22	354	284	290	284
Total cold run time: 120730 ms
Total hot run time: 40086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4420	4256	4224	4224
q2	364	273	273	273
q3	2960	2770	2750	2750
q4	1906	1630	1626	1626
q5	5286	5279	5277	5277
q6	221	130	136	130
q7	2045	1763	1747	1747
q8	3210	3295	3330	3295
q9	8393	8381	8409	8381
q10	3854	3706	3654	3654
q11	582	501	501	501
q12	778	609	600	600
q13	17537	2963	2978	2963
q14	294	277	275	275
q15	519	470	480	470
q16	481	420	432	420
q17	1759	1515	1488	1488
q18	7650	7572	7269	7269
q19	1674	1502	1617	1502
q20	1974	1800	1779	1779
q21	4970	4968	4702	4702
q22	557	495	488	488
Total cold run time: 71434 ms
Total hot run time: 53814 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

TPC-DS: Total hot run time: 172030 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7fa67a690bc24136b230258c0855f4319a338bc7, data reload: false

query1	918	372	375	372
query2	6464	1870	1843	1843
query3	6657	210	221	210
query4	28659	17459	17308	17308
query5	4182	471	487	471
query6	306	193	160	160
query7	4594	291	294	291
query8	247	187	187	187
query9	8563	2379	2374	2374
query10	463	292	288	288
query11	11627	10127	9978	9978
query12	137	80	79	79
query13	1633	360	361	360
query14	10311	7628	7776	7628
query15	215	165	163	163
query16	7760	312	319	312
query17	1640	540	512	512
query18	1806	274	269	269
query19	190	147	149	147
query20	91	84	82	82
query21	201	125	128	125
query22	4275	4053	4027	4027
query23	33775	33192	32978	32978
query24	11243	2809	2785	2785
query25	627	359	372	359
query26	1472	147	145	145
query27	2881	270	274	270
query28	7383	1994	1976	1976
query29	962	631	610	610
query30	284	151	145	145
query31	968	718	745	718
query32	96	52	55	52
query33	766	300	320	300
query34	908	496	498	496
query35	731	566	554	554
query36	1116	975	928	928
query37	148	78	78	78
query38	2848	2751	2739	2739
query39	843	801	823	801
query40	280	121	119	119
query41	47	45	43	43
query42	118	99	99	99
query43	494	472	471	471
query44	1202	718	732	718
query45	191	162	163	162
query46	1085	716	763	716
query47	1838	1792	1788	1788
query48	354	292	283	283
query49	1091	406	403	403
query50	773	411	398	398
query51	6916	6783	6810	6783
query52	102	90	93	90
query53	363	297	295	295
query54	1043	436	445	436
query55	76	73	72	72
query56	296	267	307	267
query57	1157	1059	1058	1058
query58	263	245	255	245
query59	3044	2741	2756	2741
query60	302	269	280	269
query61	99	95	97	95
query62	854	642	651	642
query63	321	296	286	286
query64	10646	2251	1774	1774
query65	3200	3112	3117	3112
query66	1394	330	331	330
query67	15268	14873	15131	14873
query68	4573	540	553	540
query69	484	329	323	323
query70	1113	1061	1128	1061
query71	374	282	276	276
query72	7093	5476	5155	5155
query73	739	331	328	328
query74	6164	5707	5659	5659
query75	3388	2651	2681	2651
query76	2689	905	968	905
query77	509	311	303	303
query78	9507	9041	8962	8962
query79	2423	527	534	527
query80	2211	522	464	464
query81	587	223	218	218
query82	862	140	129	129
query83	294	164	163	163
query84	276	85	85	85
query85	2199	326	298	298
query86	481	334	312	312
query87	3271	3084	3138	3084
query88	4455	2420	2367	2367
query89	488	380	372	372
query90	1849	195	200	195
query91	131	99	98	98
query92	59	49	48	48
query93	2620	512	501	501
query94	1274	215	215	215
query95	406	318	320	318
query96	601	278	277	277
query97	3216	3049	3050	3049
query98	218	203	195	195
query99	1590	1264	1272	1264
Total cold run time: 286951 ms
Total hot run time: 172030 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 17, 2024

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7fa67a690bc24136b230258c0855f4319a338bc7, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.06	0.06
query5	0.48	0.48	0.48
query6	1.14	0.72	0.72
query7	0.03	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.51	0.49
query10	0.54	0.53	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.78	0.79
query15	0.84	0.82	0.82
query16	0.38	0.37	0.36
query17	0.97	0.98	0.97
query18	0.22	0.21	0.23
query19	1.81	1.73	1.71
query20	0.01	0.01	0.01
query21	15.41	0.73	0.66
query22	4.11	6.98	1.97
query23	18.31	1.40	1.18
query24	2.06	0.25	0.22
query25	0.16	0.09	0.09
query26	0.30	0.21	0.22
query27	0.45	0.23	0.23
query28	13.23	1.03	1.01
query29	12.66	3.39	3.34
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.47	0.47
query33	2.89	2.92	2.88
query34	17.21	4.32	4.38
query35	4.42	4.47	4.44
query36	0.65	0.47	0.47
query37	0.19	0.16	0.15
query38	0.17	0.15	0.16
query39	0.05	0.03	0.03
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.75 s
Total hot run time: 30.65 s

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 17, 2024

PR approved by anyone and no changes requested.

@Hastyshell Hastyshell force-pushed the fix-delete-sign-with-cumu-delete branch from 7fa67a6 to a46813a Compare July 18, 2024 11:16
@Hastyshell
Copy link
Copy Markdown
Collaborator Author

Hastyshell commented Jul 18, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 18, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 18, 2024

TPC-H: Total hot run time: 39899 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a46813ab28b28abb92184b952eac3aa5534478a9, data reload: false

------ Round 1 ----------------------------------
q1	17741	5195	4283	4283
q2	2023	200	196	196
q3	10492	1166	1104	1104
q4	10187	839	767	767
q5	7538	2703	2606	2606
q6	223	134	137	134
q7	951	602	609	602
q8	9212	2060	2097	2060
q9	8691	6538	6566	6538
q10	8762	3783	3790	3783
q11	466	238	232	232
q12	428	233	222	222
q13	17774	2967	3007	2967
q14	270	231	247	231
q15	527	486	501	486
q16	504	386	378	378
q17	967	727	702	702
q18	8143	7480	7521	7480
q19	7450	1396	1352	1352
q20	674	319	338	319
q21	5047	3174	3202	3174
q22	349	290	283	283
Total cold run time: 118419 ms
Total hot run time: 39899 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4376	4245	4223	4223
q2	397	295	273	273
q3	3045	2942	2971	2942
q4	1999	1775	1736	1736
q5	5568	5603	5555	5555
q6	233	135	135	135
q7	2333	1915	1960	1915
q8	3327	3494	3468	3468
q9	8791	8832	8784	8784
q10	4144	3756	3865	3756
q11	604	497	486	486
q12	822	682	630	630
q13	17004	3181	3180	3180
q14	316	312	283	283
q15	531	484	483	483
q16	512	422	430	422
q17	1841	1512	1516	1512
q18	8048	7923	7911	7911
q19	1745	1663	1564	1564
q20	2914	1907	1831	1831
q21	5083	4918	4779	4779
q22	623	511	495	495
Total cold run time: 74256 ms
Total hot run time: 56363 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 18, 2024

TPC-DS: Total hot run time: 173777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a46813ab28b28abb92184b952eac3aa5534478a9, data reload: false

query1	924	368	368	368
query2	6459	2003	1933	1933
query3	6652	208	218	208
query4	27117	17534	17394	17394
query5	3810	475	483	475
query6	283	165	156	156
query7	4581	296	285	285
query8	233	187	195	187
query9	8443	2517	2498	2498
query10	438	291	292	291
query11	13055	10018	10109	10018
query12	111	85	84	84
query13	1638	375	373	373
query14	10341	7957	7282	7282
query15	232	173	178	173
query16	7719	331	331	331
query17	1341	560	540	540
query18	1804	286	286	286
query19	204	158	159	158
query20	89	83	82	82
query21	205	129	130	129
query22	4244	3962	4045	3962
query23	33895	33958	33527	33527
query24	11015	2943	2958	2943
query25	668	399	383	383
query26	749	157	150	150
query27	2256	283	285	283
query28	5783	2125	2124	2124
query29	886	657	617	617
query30	266	154	150	150
query31	944	755	757	755
query32	96	56	56	56
query33	667	328	298	298
query34	875	503	497	497
query35	684	606	603	603
query36	1156	982	971	971
query37	140	81	77	77
query38	2961	2845	2853	2845
query39	911	803	820	803
query40	196	119	119	119
query41	45	43	43	43
query42	132	103	100	100
query43	520	471	499	471
query44	1087	726	733	726
query45	194	163	164	163
query46	1076	749	703	703
query47	1841	1778	1762	1762
query48	370	290	296	290
query49	838	402	409	402
query50	782	402	394	394
query51	6962	6737	6710	6710
query52	104	96	92	92
query53	363	295	297	295
query54	847	466	459	459
query55	78	74	76	74
query56	289	275	274	274
query57	1128	1014	1067	1014
query58	243	235	257	235
query59	2885	2780	2758	2758
query60	309	268	277	268
query61	121	95	98	95
query62	770	670	643	643
query63	317	299	291	291
query64	9184	2209	1660	1660
query65	3153	3102	3090	3090
query66	783	333	331	331
query67	15673	15188	14898	14898
query68	4660	560	545	545
query69	670	425	355	355
query70	1189	1129	1120	1120
query71	413	293	283	283
query72	8391	5722	5615	5615
query73	764	328	330	328
query74	6147	5764	5702	5702
query75	3730	2656	2653	2653
query76	2882	908	926	908
query77	724	316	317	316
query78	9690	9071	10089	9071
query79	2889	535	538	535
query80	2230	473	520	473
query81	598	220	222	220
query82	588	133	144	133
query83	299	173	166	166
query84	260	90	101	90
query85	1623	327	299	299
query86	451	318	307	307
query87	3302	3086	3227	3086
query88	3776	2487	2453	2453
query89	469	387	379	379
query90	1904	189	190	189
query91	131	103	100	100
query92	59	48	53	48
query93	2702	518	523	518
query94	1293	221	213	213
query95	403	327	326	326
query96	596	320	274	274
query97	3197	3065	3017	3017
query98	228	199	195	195
query99	1475	1278	1205	1205
Total cold run time: 281898 ms
Total hot run time: 173777 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 18, 2024

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a46813ab28b28abb92184b952eac3aa5534478a9, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.05
query3	0.23	0.05	0.06
query4	1.68	0.08	0.08
query5	0.51	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.55	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.58
query14	0.76	0.78	0.77
query15	0.86	0.80	0.80
query16	0.38	0.37	0.38
query17	0.99	1.00	0.99
query18	0.23	0.21	0.23
query19	1.96	1.82	1.79
query20	0.02	0.01	0.01
query21	15.38	0.74	0.66
query22	4.24	7.13	1.92
query23	18.33	1.38	1.28
query24	2.08	0.23	0.22
query25	0.17	0.09	0.08
query26	0.30	0.21	0.20
query27	0.45	0.24	0.23
query28	13.31	1.01	1.00
query29	12.64	3.36	3.28
query30	0.25	0.06	0.07
query31	2.86	0.39	0.39
query32	3.26	0.46	0.48
query33	2.85	2.85	2.90
query34	16.93	4.37	4.40
query35	4.42	4.46	4.44
query36	0.66	0.47	0.49
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.14	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.78 s
Total hot run time: 30.76 s

dataroaring
dataroaring approved these changes Jul 18, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 18, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 9487945 into apache:master Jul 19, 2024
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
@Hastyshell @dataroaring
[fix](delete-sign) Disable deletion with delete sign when doing cumul…
91293d5 
…ative compaction (#37950)

## Proposed changes

Delete sign could not be applied when delete on cumu compaction is
enabled, bucause it is meant for delete with predicates. If delete
design is applied on cumu compaction, it will lose effect when doing
base compaction.

So disable delete sign when doing cumulative compaction.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luwei16 luwei16 luwei16 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Hastyshell @doris-robot @luwei16 @dataroaring