[opt](load) limit concurrency of delete_bitmap computation by kaijchen · Pull Request #52962 · apache/doris

kaijchen · 2025-07-08T12:42:19Z

What problem does this PR solve?

Related PR: #48156

Problem Summary:

Previously, the same thread pool was used to calculate the DeleteBitmap during both the build rowset and publish stages. This led to high CPU usage and resource contention, especially impacting memtable flushing performance.

To mitigate this, a dedicated thread pool is introduced for DeleteBitmap computation during the build rowset stage. This new pool is configured with a lower maximum thread count (8 vs. the original 32) to better control resource consumption and reduce interference with other critical operations.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-07-08T12:42:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

kaijchen · 2025-07-08T12:42:28Z

run buildall

kaijchen · 2025-07-08T13:07:30Z

run buildall

kaijchen · 2025-07-08T13:17:11Z

run buildall

doris-robot · 2025-07-08T15:03:42Z

TPC-H: Total hot run time: 34367 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24a8499c4d8779888afb3884f75a30d245c30cf1, data reload: false

------ Round 1 ----------------------------------
q1	17600	5459	5247	5247
q2	1941	293	190	190
q3	10398	1473	746	746
q4	10263	1113	532	532
q5	7955	2673	2682	2673
q6	205	175	133	133
q7	998	766	607	607
q8	9309	1501	1250	1250
q9	7499	5330	5343	5330
q10	6949	2435	1955	1955
q11	523	289	273	273
q12	353	389	221	221
q13	17796	3827	3086	3086
q14	244	246	218	218
q15	562	485	478	478
q16	440	437	385	385
q17	599	938	371	371
q18	7513	7258	7083	7083
q19	1218	1092	573	573
q20	338	387	224	224
q21	4239	3374	2497	2497
q22	369	333	295	295
Total cold run time: 107311 ms
Total hot run time: 34367 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5467	5462	5461	5461
q2	256	361	217	217
q3	2200	2757	2325	2325
q4	1435	1856	1353	1353
q5	4688	4469	4585	4469
q6	249	182	126	126
q7	2193	1982	1844	1844
q8	2942	2925	2953	2925
q9	7424	7309	7403	7309
q10	3305	3343	3016	3016
q11	637	524	492	492
q12	721	827	644	644
q13	3751	4021	3337	3337
q14	288	315	293	293
q15	549	496	479	479
q16	466	539	456	456
q17	1294	1868	1510	1510
q18	8109	7653	6949	6949
q19	843	781	898	781
q20	1983	1989	1807	1807
q21	5077	4573	4413	4413
q22	695	580	562	562
Total cold run time: 54572 ms
Total hot run time: 50768 ms

doris-robot · 2025-07-08T15:15:12Z

TPC-DS: Total hot run time: 185970 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24a8499c4d8779888afb3884f75a30d245c30cf1, data reload: false

query1	999	403	409	403
query2	6562	1706	1631	1631
query3	6742	211	209	209
query4	26095	23361	22970	22970
query5	4392	586	451	451
query6	317	233	211	211
query7	4624	506	295	295
query8	274	234	240	234
query9	8627	2698	2692	2692
query10	488	337	280	280
query11	16002	15101	14995	14995
query12	166	107	103	103
query13	1659	527	409	409
query14	9804	6046	6015	6015
query15	207	187	176	176
query16	7656	426	258	258
query17	1421	686	588	588
query18	2024	395	296	296
query19	190	183	154	154
query20	122	112	113	112
query21	212	125	103	103
query22	4056	4288	4077	4077
query23	33912	33020	32916	32916
query24	8418	2423	2399	2399
query25	504	481	394	394
query26	744	276	154	154
query27	2723	504	346	346
query28	4334	2153	2134	2134
query29	650	604	444	444
query30	288	218	193	193
query31	923	823	772	772
query32	70	66	60	60
query33	562	352	303	303
query34	838	849	544	544
query35	632	660	566	566
query36	951	972	899	899
query37	130	105	85	85
query38	4191	4232	4107	4107
query39	1480	1421	1406	1406
query40	214	121	109	109
query41	58	55	51	51
query42	125	110	107	107
query43	508	494	475	475
query44	1412	861	831	831
query45	178	173	168	168
query46	869	1023	631	631
query47	1738	1781	1717	1717
query48	384	417	317	317
query49	684	471	386	386
query50	683	683	423	423
query51	4172	4150	4140	4140
query52	116	107	102	102
query53	244	265	189	189
query54	575	571	501	501
query55	80	82	83	82
query56	311	312	313	312
query57	1194	1185	1142	1142
query58	269	270	260	260
query59	2595	2672	2551	2551
query60	319	338	299	299
query61	126	131	134	131
query62	793	725	669	669
query63	231	197	194	194
query64	3318	1164	823	823
query65	4286	4270	4187	4187
query66	878	409	338	338
query67	15546	15536	15173	15173
query68	6858	887	528	528
query69	524	320	267	267
query70	1210	1144	1112	1112
query71	415	330	315	315
query72	5757	4770	4788	4770
query73	644	619	351	351
query74	9280	8871	8970	8871
query75	3211	3211	2702	2702
query76	3149	1150	766	766
query77	486	392	308	308
query78	11051	10965	10177	10177
query79	1046	846	620	620
query80	595	552	496	496
query81	480	263	232	232
query82	211	134	100	100
query83	271	264	246	246
query84	252	114	100	100
query85	855	351	320	320
query86	337	293	292	292
query87	4453	4419	4270	4270
query88	2952	2319	2278	2278
query89	386	317	299	299
query90	1723	215	214	214
query91	209	158	108	108
query92	65	63	56	56
query93	1021	965	593	593
query94	620	318	201	201
query95	376	298	291	291
query96	499	574	280	280
query97	2679	2775	2650	2650
query98	230	207	205	205
query99	1359	1391	1260	1260
Total cold run time: 269099 ms
Total hot run time: 185970 ms

doris-robot · 2025-07-08T15:20:29Z

ClickBench: Total hot run time: 29.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24a8499c4d8779888afb3884f75a30d245c30cf1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.08	0.08
query4	1.62	0.10	0.11
query5	0.43	0.43	0.41
query6	1.16	0.67	0.66
query7	0.04	0.02	0.02
query8	0.05	0.03	0.04
query9	0.61	0.52	0.51
query10	0.57	0.56	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.60	0.62
query14	0.78	0.84	0.80
query15	0.91	0.89	0.90
query16	0.40	0.38	0.39
query17	1.06	1.07	1.05
query18	0.23	0.22	0.21
query19	2.06	1.88	1.90
query20	0.02	0.01	0.01
query21	15.40	0.91	0.54
query22	0.76	1.13	0.69
query23	14.94	1.40	0.62
query24	6.63	2.28	0.75
query25	0.51	0.18	0.10
query26	0.65	0.17	0.15
query27	0.07	0.05	0.05
query28	9.01	0.94	0.45
query29	12.63	3.97	3.37
query30	0.25	0.10	0.06
query31	2.84	0.63	0.40
query32	3.24	0.57	0.50
query33	3.03	3.11	3.17
query34	15.97	5.42	4.79
query35	4.82	4.92	4.86
query36	0.69	0.49	0.48
query37	0.09	0.07	0.07
query38	0.05	0.03	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.16 s
Total hot run time: 29.73 s

doris-robot · 2025-07-08T16:31:07Z

BE UT Coverage Report

Increment line coverage 76.92% (10/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.33% (15508/27051)
Line Coverage	46.30% (140920/304340)
Region Coverage	45.56% (71253/156402)
Branch Coverage	40.29% (37554/93220)

kaijchen · 2025-08-14T09:41:02Z

run buildall

doris-robot · 2025-08-14T11:13:00Z

TPC-H: Total hot run time: 33874 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f3c87c2963971a0221122a4cb044b78e676abf4, data reload: false

------ Round 1 ----------------------------------
q1	17646	5329	5023	5023
q2	1910	278	189	189
q3	10337	1278	735	735
q4	10251	1004	546	546
q5	8344	2361	2372	2361
q6	193	159	133	133
q7	893	739	595	595
q8	9302	1291	1079	1079
q9	7290	5092	5190	5092
q10	6987	2386	1969	1969
q11	507	284	273	273
q12	368	351	213	213
q13	17781	3682	3049	3049
q14	231	238	228	228
q15	568	489	483	483
q16	421	420	378	378
q17	608	853	364	364
q18	7696	7067	7137	7067
q19	1479	949	565	565
q20	336	329	219	219
q21	3886	3210	2320	2320
q22	1048	1031	993	993
Total cold run time: 108082 ms
Total hot run time: 33874 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5298	5074	5161	5074
q2	233	322	216	216
q3	2198	2689	2318	2318
q4	1341	1804	1328	1328
q5	4425	4404	4564	4404
q6	225	178	136	136
q7	2078	2011	1758	1758
q8	2623	2485	2568	2485
q9	7424	7265	7213	7213
q10	3169	3257	3104	3104
q11	578	522	499	499
q12	718	729	632	632
q13	3573	3898	3300	3300
q14	285	452	300	300
q15	517	483	476	476
q16	440	479	432	432
q17	1181	1580	1335	1335
q18	7810	7664	7757	7664
q19	844	827	872	827
q20	1901	2020	1808	1808
q21	4825	4243	4379	4243
q22	1104	1049	987	987
Total cold run time: 52790 ms
Total hot run time: 50539 ms

doris-robot · 2025-08-14T11:24:33Z

TPC-DS: Total hot run time: 184421 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f3c87c2963971a0221122a4cb044b78e676abf4, data reload: false

query1	987	387	407	387
query2	6531	1721	1739	1721
query3	6741	222	222	222
query4	26997	23285	22857	22857
query5	4346	628	484	484
query6	309	220	213	213
query7	4628	510	285	285
query8	279	240	210	210
query9	8591	2856	2818	2818
query10	457	337	279	279
query11	15797	15001	14718	14718
query12	166	114	112	112
query13	1654	531	400	400
query14	9081	5681	5807	5681
query15	209	183	165	165
query16	7193	669	482	482
query17	1217	711	605	605
query18	2005	422	325	325
query19	199	194	171	171
query20	127	121	119	119
query21	214	127	112	112
query22	4115	4251	3995	3995
query23	34699	33415	33360	33360
query24	8225	2339	2394	2339
query25	572	559	396	396
query26	896	270	160	160
query27	2709	511	331	331
query28	4359	2238	2197	2197
query29	720	562	439	439
query30	283	217	191	191
query31	883	794	694	694
query32	82	72	72	72
query33	540	365	392	365
query34	796	834	513	513
query35	786	846	741	741
query36	1006	1042	921	921
query37	129	126	82	82
query38	4119	3995	4072	3995
query39	1464	1426	1394	1394
query40	230	125	117	117
query41	59	60	55	55
query42	125	108	110	108
query43	504	504	453	453
query44	1350	849	846	846
query45	177	170	172	170
query46	852	999	636	636
query47	1758	1871	1716	1716
query48	380	411	306	306
query49	695	486	395	395
query50	638	686	387	387
query51	4095	4330	4083	4083
query52	115	113	102	102
query53	244	265	192	192
query54	591	596	522	522
query55	91	85	88	85
query56	309	312	301	301
query57	1180	1168	1147	1147
query58	275	273	269	269
query59	2691	2644	2680	2644
query60	357	355	337	337
query61	132	127	126	126
query62	814	712	667	667
query63	220	194	187	187
query64	3535	1068	711	711
query65	4318	4199	4233	4199
query66	1029	434	331	331
query67	15546	15369	15038	15038
query68	7970	927	580	580
query69	486	324	286	286
query70	1224	1154	1136	1136
query71	461	330	308	308
query72	5689	4724	4820	4724
query73	734	634	352	352
query74	8878	9056	8912	8912
query75	3805	3062	2629	2629
query76	3642	1143	738	738
query77	795	394	313	313
query78	9639	9595	8777	8777
query79	2735	831	595	595
query80	694	550	465	465
query81	497	250	219	219
query82	458	139	104	104
query83	296	256	241	241
query84	303	104	85	85
query85	819	438	356	356
query86	393	316	315	315
query87	4354	4276	4157	4157
query88	3355	2191	2162	2162
query89	383	318	278	278
query90	1823	226	215	215
query91	139	145	109	109
query92	90	70	66	66
query93	1890	982	634	634
query94	667	403	312	312
query95	395	312	295	295
query96	490	579	270	270
query97	2641	2659	2599	2599
query98	235	227	218	218
query99	1443	1399	1290	1290
Total cold run time: 273844 ms
Total hot run time: 184421 ms

doris-robot · 2025-08-14T11:30:03Z

ClickBench: Total hot run time: 32.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f3c87c2963971a0221122a4cb044b78e676abf4, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.25	0.07	0.07
query4	1.64	0.11	0.11
query5	0.43	0.40	0.42
query6	1.17	0.64	0.65
query7	0.03	0.02	0.01
query8	0.05	0.03	0.03
query9	0.63	0.52	0.52
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.64	0.61	0.61
query14	0.79	0.82	0.85
query15	0.91	0.85	0.87
query16	0.39	0.39	0.39
query17	1.03	1.04	1.05
query18	0.21	0.19	0.20
query19	1.94	1.86	1.78
query20	0.01	0.01	0.02
query21	15.42	0.91	0.55
query22	0.74	1.15	0.71
query23	15.00	1.39	0.66
query24	6.74	1.69	1.02
query25	0.52	0.25	0.05
query26	0.54	0.15	0.13
query27	0.07	0.05	0.05
query28	10.42	0.94	0.42
query29	12.60	4.00	3.28
query30	3.06	2.97	3.00
query31	2.82	0.59	0.37
query32	3.24	0.56	0.46
query33	3.02	3.17	3.14
query34	16.11	5.47	4.90
query35	4.92	4.96	4.94
query36	0.70	0.52	0.49
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.03	0.02	0.02
Total cold run time: 107.58 s
Total hot run time: 32.81 s

doris-robot · 2025-08-14T12:36:17Z

BE UT Coverage Report

Increment line coverage 76.92% (10/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.49% (16785/28217)
Line Coverage	48.42% (152607/315154)
Region Coverage	37.37% (115958/310275)
Branch Coverage	40.30% (51184/127011)

hello-stephen · 2025-08-14T13:39:03Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (13/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.89% (22640/27647)
Line Coverage	74.60% (234941/314928)
Region Coverage	61.92% (195381/315516)
Branch Coverage	66.05% (84513/127944)

dataroaring · 2025-09-08T08:55:06Z

run buildall

doris-robot · 2025-09-08T11:44:45Z

TPC-H: Total hot run time: 34858 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78feb2a1fe5d32a581b934ec5ee434aacae617b1, data reload: false

------ Round 1 ----------------------------------
q1	17657	5296	5086	5086
q2	2043	345	219	219
q3	10203	1350	724	724
q4	10229	1032	533	533
q5	7546	2466	2413	2413
q6	191	171	135	135
q7	963	801	630	630
q8	9340	1346	1158	1158
q9	7009	5130	5168	5130
q10	6988	2404	1967	1967
q11	495	302	287	287
q12	384	377	224	224
q13	17788	3709	3061	3061
q14	245	254	230	230
q15	585	500	487	487
q16	1014	1006	946	946
q17	611	875	381	381
q18	7998	7109	7004	7004
q19	1105	977	569	569
q20	367	350	240	240
q21	4086	3253	2449	2449
q22	1074	1058	985	985
Total cold run time: 107921 ms
Total hot run time: 34858 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5214	5062	5157	5062
q2	252	332	233	233
q3	2200	2636	2304	2304
q4	1363	1800	1401	1401
q5	4319	4712	4637	4637
q6	229	173	133	133
q7	2036	1992	1818	1818
q8	2706	2666	2617	2617
q9	7538	7372	7628	7372
q10	3076	3259	2905	2905
q11	597	527	525	525
q12	700	790	673	673
q13	3557	3925	3409	3409
q14	299	301	279	279
q15	607	510	504	504
q16	1141	1130	1046	1046
q17	1193	1658	1398	1398
q18	7971	7650	7774	7650
q19	842	873	870	870
q20	1989	2054	1987	1987
q21	5166	4477	4524	4477
q22	1124	1032	966	966
Total cold run time: 54119 ms
Total hot run time: 52266 ms

doris-robot · 2025-09-08T11:56:30Z

TPC-DS: Total hot run time: 188774 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78feb2a1fe5d32a581b934ec5ee434aacae617b1, data reload: false

query1	1060	444	441	441
query2	6558	1686	1679	1679
query3	6751	225	227	225
query4	26147	23652	22894	22894
query5	4364	637	546	546
query6	356	251	244	244
query7	4654	524	300	300
query8	307	262	275	262
query9	8635	2956	2948	2948
query10	502	368	296	296
query11	15652	15170	14820	14820
query12	178	121	121	121
query13	1694	552	415	415
query14	11208	9211	9165	9165
query15	210	194	171	171
query16	7512	740	508	508
query17	1214	752	626	626
query18	2067	434	369	369
query19	203	225	176	176
query20	132	128	122	122
query21	210	140	115	115
query22	4152	4171	4055	4055
query23	33856	32832	32964	32832
query24	8155	2410	2406	2406
query25	583	522	462	462
query26	1217	285	171	171
query27	2733	509	361	361
query28	4386	2282	2265	2265
query29	852	617	504	504
query30	293	225	198	198
query31	923	797	742	742
query32	99	85	90	85
query33	568	403	403	403
query34	809	856	519	519
query35	841	825	769	769
query36	987	1017	923	923
query37	132	128	96	96
query38	3519	3514	3463	3463
query39	1492	1445	1426	1426
query40	223	138	131	131
query41	65	64	63	63
query42	136	116	124	116
query43	492	511	486	486
query44	1402	876	880	876
query45	188	173	175	173
query46	875	1033	660	660
query47	1771	1807	1716	1716
query48	402	428	323	323
query49	771	507	424	424
query50	659	717	411	411
query51	3916	3910	3996	3910
query52	118	115	109	109
query53	256	284	199	199
query54	626	619	560	560
query55	99	93	90	90
query56	381	361	330	330
query57	1229	1243	1140	1140
query58	299	285	287	285
query59	2601	2714	2630	2630
query60	370	354	346	346
query61	172	171	159	159
query62	829	733	652	652
query63	243	202	209	202
query64	4475	1189	867	867
query65	4042	3938	3989	3938
query66	1135	457	368	368
query67	15691	15230	14943	14943
query68	6924	940	591	591
query69	510	347	317	317
query70	1353	1288	1327	1288
query71	578	363	335	335
query72	6297	5214	4942	4942
query73	654	585	362	362
query74	8941	9114	8867	8867
query75	3314	3222	2783	2783
query76	3226	1145	767	767
query77	488	406	333	333
query78	9654	9713	8839	8839
query79	2888	843	590	590
query80	756	701	534	534
query81	514	266	231	231
query82	651	161	136	136
query83	266	265	253	253
query84	266	120	97	97
query85	910	494	458	458
query86	395	337	315	315
query87	3709	3738	3574	3574
query88	3706	2207	2240	2207
query89	397	327	308	308
query90	1911	242	228	228
query91	161	169	137	137
query92	148	76	73	73
query93	2538	1009	642	642
query94	747	426	328	328
query95	425	350	335	335
query96	497	572	284	284
query97	2965	3039	2878	2878
query98	258	221	215	215
query99	1472	1416	1297	1297
Total cold run time: 276196 ms
Total hot run time: 188774 ms

doris-robot · 2025-09-08T12:01:58Z

ClickBench: Total hot run time: 29.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78feb2a1fe5d32a581b934ec5ee434aacae617b1, data reload: false

query1	0.06	0.04	0.05
query2	0.09	0.05	0.06
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.28	0.26	0.25
query6	1.20	0.65	0.64
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.63	0.53	0.51
query10	0.59	0.57	0.59
query11	0.17	0.11	0.12
query12	0.15	0.13	0.12
query13	0.64	0.63	0.62
query14	1.03	1.05	1.02
query15	0.86	0.88	0.86
query16	0.40	0.40	0.43
query17	1.05	1.05	1.03
query18	0.24	0.20	0.20
query19	1.94	1.82	1.80
query20	0.01	0.02	0.01
query21	15.43	0.97	0.58
query22	0.80	1.27	0.66
query23	14.82	1.40	0.63
query24	6.93	1.03	0.83
query25	0.47	0.15	0.11
query26	0.58	0.17	0.13
query27	0.06	0.06	0.06
query28	9.66	0.91	0.44
query29	12.67	3.97	3.22
query30	0.29	0.13	0.11
query31	2.82	0.60	0.38
query32	3.23	0.56	0.48
query33	3.12	3.08	3.18
query34	16.08	5.47	4.84
query35	4.92	4.96	4.88
query36	0.70	0.51	0.50
query37	0.10	0.07	0.07
query38	0.07	0.04	0.05
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.04
query43	0.05	0.04	0.04
Total cold run time: 104.44 s
Total hot run time: 29.82 s

… calculation

kaijchen · 2025-09-10T11:28:02Z

run buildall

kaijchen · 2025-09-10T13:06:14Z

run buildall

doris-robot · 2025-09-10T14:14:40Z

TPC-H: Total hot run time: 34912 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13a0b35b354e6e7a3d9e57af9586f2de9e12304e, data reload: false

------ Round 1 ----------------------------------
q1	17661	5244	5059	5059
q2	2018	322	249	249
q3	10229	1316	742	742
q4	10222	1056	528	528
q5	7507	2543	2304	2304
q6	195	172	137	137
q7	970	778	645	645
q8	9368	1416	1158	1158
q9	6962	5141	5157	5141
q10	6897	2397	1999	1999
q11	518	302	288	288
q12	357	368	239	239
q13	17781	3668	3057	3057
q14	253	230	217	217
q15	549	488	491	488
q16	1001	1013	934	934
q17	603	883	369	369
q18	7468	7267	7135	7135
q19	1114	984	595	595
q20	360	360	237	237
q21	3956	3218	2406	2406
q22	1090	1035	985	985
Total cold run time: 107079 ms
Total hot run time: 34912 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5151	5132	5096	5096
q2	249	336	231	231
q3	2224	2705	2306	2306
q4	1344	1830	1373	1373
q5	4290	4668	4530	4530
q6	219	180	132	132
q7	2039	2026	1866	1866
q8	2685	2628	2618	2618
q9	7302	7314	7259	7259
q10	3186	3291	2918	2918
q11	585	529	526	526
q12	698	821	674	674
q13	3513	3914	3341	3341
q14	287	306	296	296
q15	536	496	547	496
q16	1086	1122	1079	1079
q17	1182	1624	1411	1411
q18	8053	7924	7560	7560
q19	817	796	855	796
q20	2014	2087	1969	1969
q21	5077	4495	4298	4298
q22	1071	1047	1012	1012
Total cold run time: 53608 ms
Total hot run time: 51787 ms

doris-robot · 2025-09-10T14:26:29Z

TPC-DS: Total hot run time: 189531 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13a0b35b354e6e7a3d9e57af9586f2de9e12304e, data reload: false

query1	1044	484	418	418
query2	6558	1719	1795	1719
query3	6753	226	231	226
query4	26533	23605	23151	23151
query5	4706	656	489	489
query6	351	232	228	228
query7	4668	528	303	303
query8	316	262	262	262
query9	8560	2898	2915	2898
query10	457	354	303	303
query11	15752	15153	14723	14723
query12	176	120	116	116
query13	1686	545	417	417
query14	11218	9240	9144	9144
query15	210	196	172	172
query16	7550	713	483	483
query17	1186	744	628	628
query18	2040	435	328	328
query19	208	200	183	183
query20	140	123	121	121
query21	208	129	122	122
query22	4195	4334	4102	4102
query23	33936	33353	33127	33127
query24	8209	2409	2407	2407
query25	568	514	442	442
query26	1242	278	172	172
query27	2731	508	374	374
query28	4401	2281	2251	2251
query29	779	632	482	482
query30	304	222	201	201
query31	924	821	762	762
query32	87	85	80	80
query33	569	385	379	379
query34	798	866	532	532
query35	838	838	776	776
query36	974	1026	911	911
query37	130	118	106	106
query38	3557	3513	3473	3473
query39	1519	1430	1462	1430
query40	236	144	130	130
query41	71	71	67	67
query42	129	121	122	121
query43	530	498	469	469
query44	1357	865	858	858
query45	187	185	171	171
query46	864	1031	661	661
query47	1755	1821	1742	1742
query48	393	441	326	326
query49	780	536	439	439
query50	645	691	414	414
query51	3936	4005	3946	3946
query52	121	117	107	107
query53	249	275	200	200
query54	650	648	582	582
query55	100	98	96	96
query56	367	399	327	327
query57	1193	1206	1134	1134
query58	294	289	281	281
query59	2545	2710	2471	2471
query60	355	361	341	341
query61	167	161	160	160
query62	885	762	699	699
query63	228	203	194	194
query64	4409	1192	826	826
query65	4063	3966	3949	3949
query66	1118	447	357	357
query67	15515	15305	15137	15137
query68	9075	978	583	583
query69	491	337	286	286
query70	1383	1288	1270	1270
query71	576	351	326	326
query72	6037	5197	5304	5197
query73	791	697	368	368
query74	9251	9193	8962	8962
query75	4436	3292	2745	2745
query76	3730	1156	733	733
query77	897	408	331	331
query78	9661	9754	8889	8889
query79	2653	855	585	585
query80	718	585	517	517
query81	479	262	234	234
query82	344	170	132	132
query83	292	261	245	245
query84	305	112	96	96
query85	855	468	427	427
query86	365	340	297	297
query87	3756	3745	3720	3720
query88	2799	2227	2225	2225
query89	430	335	289	289
query90	2077	232	230	230
query91	163	162	136	136
query92	93	76	69	69
query93	2100	1007	653	653
query94	689	426	322	322
query95	412	339	339	339
query96	478	605	284	284
query97	2933	2997	2884	2884
query98	250	224	211	211
query99	1409	1438	1300	1300
Total cold run time: 279151 ms
Total hot run time: 189531 ms

doris-robot · 2025-09-10T14:31:57Z

ClickBench: Total hot run time: 29.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 13a0b35b354e6e7a3d9e57af9586f2de9e12304e, data reload: false

query1	0.06	0.04	0.05
query2	0.10	0.05	0.06
query3	0.25	0.08	0.08
query4	1.60	0.12	0.12
query5	0.27	0.26	0.26
query6	1.16	0.66	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.61	0.52	0.52
query10	0.58	0.58	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.63	0.62
query14	1.02	1.04	1.01
query15	0.87	0.85	0.87
query16	0.40	0.39	0.39
query17	1.06	1.05	1.07
query18	0.21	0.20	0.20
query19	1.90	1.84	1.78
query20	0.02	0.01	0.01
query21	15.40	0.96	0.58
query22	0.76	1.25	0.80
query23	14.78	1.37	0.64
query24	7.28	0.94	0.76
query25	0.49	0.12	0.25
query26	0.65	0.17	0.13
query27	0.08	0.05	0.05
query28	9.90	0.87	0.42
query29	12.55	3.90	3.25
query30	0.29	0.14	0.11
query31	2.83	0.58	0.40
query32	3.23	0.56	0.48
query33	3.05	3.17	3.06
query34	16.00	5.46	4.81
query35	4.93	4.92	4.97
query36	0.72	0.52	0.51
query37	0.10	0.07	0.08
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 104.62 s
Total hot run time: 29.9 s

hello-stephen · 2025-09-10T15:55:59Z

BE UT Coverage Report

Increment line coverage 64.71% (11/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.97% (17303/33296)
Line Coverage	37.31% (157689/422661)
Region Coverage	31.95% (120310/376564)
Branch Coverage	33.33% (52833/158511)

hello-stephen · 2025-09-10T16:47:53Z

BE Regression && UT Coverage Report

Increment line coverage 94.12% (16/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.78% (23142/32695)
Line Coverage	57.14% (241359/422407)
Region Coverage	52.52% (200614/381993)
Branch Coverage	54.18% (86369/159414)

kaijchen · 2025-09-11T02:13:02Z

run buildall

doris-robot · 2025-09-11T02:47:23Z

TPC-H: Total hot run time: 34495 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 817eb80eea0f4a85ad8f7a44191a60f014e31240, data reload: false

------ Round 1 ----------------------------------
q1	17611	5218	5114	5114
q2	1998	315	203	203
q3	10290	1305	705	705
q4	10247	1022	506	506
q5	7563	2435	2338	2338
q6	182	167	136	136
q7	912	769	628	628
q8	9345	1346	1123	1123
q9	7005	5094	5098	5094
q10	6934	2401	1952	1952
q11	481	305	284	284
q12	349	358	235	235
q13	17765	3643	3024	3024
q14	253	230	209	209
q15	571	487	491	487
q16	1009	1004	933	933
q17	611	861	363	363
q18	7414	7016	7120	7016
q19	1564	966	597	597
q20	368	339	235	235
q21	3712	3185	2328	2328
q22	1063	1048	985	985
Total cold run time: 107247 ms
Total hot run time: 34495 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5241	5048	5133	5048
q2	253	327	232	232
q3	2180	2643	2293	2293
q4	1372	1750	1295	1295
q5	4226	4589	4492	4492
q6	205	163	129	129
q7	2068	1967	1867	1867
q8	2656	2626	2506	2506
q9	7486	7348	7436	7348
q10	3141	3290	2812	2812
q11	576	552	516	516
q12	694	794	871	794
q13	3455	3835	3337	3337
q14	321	318	296	296
q15	531	479	480	479
q16	1112	1107	1084	1084
q17	1163	1814	1384	1384
q18	7953	7751	7608	7608
q19	817	850	919	850
q20	2052	2066	1962	1962
q21	5006	4355	4218	4218
q22	1076	1032	1007	1007
Total cold run time: 53584 ms
Total hot run time: 51557 ms

doris-robot · 2025-09-11T02:59:04Z

TPC-DS: Total hot run time: 188487 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 817eb80eea0f4a85ad8f7a44191a60f014e31240, data reload: false

query1	1071	442	396	396
query2	6574	1670	1699	1670
query3	6786	227	226	226
query4	26474	23744	22873	22873
query5	4459	672	517	517
query6	358	256	252	252
query7	4656	515	304	304
query8	326	269	260	260
query9	8636	2947	2939	2939
query10	497	395	317	317
query11	15776	15444	14853	14853
query12	189	130	126	126
query13	1695	562	440	440
query14	11595	9048	9189	9048
query15	209	183	176	176
query16	7665	652	485	485
query17	1208	807	635	635
query18	2069	420	338	338
query19	202	198	184	184
query20	137	132	126	126
query21	217	129	118	118
query22	4046	4178	4128	4128
query23	33902	33045	32896	32896
query24	8090	2385	2393	2385
query25	572	520	466	466
query26	1247	279	177	177
query27	2720	513	353	353
query28	4392	2264	2248	2248
query29	744	623	528	528
query30	293	221	202	202
query31	950	812	722	722
query32	93	86	84	84
query33	593	390	357	357
query34	820	854	528	528
query35	811	843	759	759
query36	998	1037	938	938
query37	131	113	95	95
query38	3548	3490	3534	3490
query39	1513	1446	1409	1409
query40	232	137	127	127
query41	76	79	60	60
query42	130	118	120	118
query43	502	497	471	471
query44	1342	886	861	861
query45	194	174	167	167
query46	851	1013	642	642
query47	1784	1798	1753	1753
query48	400	425	321	321
query49	758	522	407	407
query50	653	686	410	410
query51	3914	3994	3880	3880
query52	116	120	114	114
query53	251	278	203	203
query54	629	625	550	550
query55	99	93	92	92
query56	337	378	322	322
query57	1228	1194	1111	1111
query58	290	281	283	281
query59	2600	2629	2560	2560
query60	369	357	348	348
query61	171	159	163	159
query62	836	755	675	675
query63	239	200	202	200
query64	4353	1131	831	831
query65	4126	4022	4005	4005
query66	1108	444	356	356
query67	15492	15206	15172	15172
query68	7285	924	583	583
query69	494	347	292	292
query70	1391	1285	1250	1250
query71	540	356	321	321
query72	5510	5068	5065	5065
query73	655	642	364	364
query74	8886	9078	8627	8627
query75	3303	3245	2717	2717
query76	3273	1171	744	744
query77	529	408	333	333
query78	9409	9678	8852	8852
query79	2474	803	589	589
query80	687	639	525	525
query81	514	258	237	237
query82	215	164	139	139
query83	271	284	255	255
query84	257	116	92	92
query85	850	467	422	422
query86	380	342	319	319
query87	3703	3807	3628	3628
query88	3079	2216	2226	2216
query89	390	315	296	296
query90	1957	246	230	230
query91	215	168	132	132
query92	87	86	72	72
query93	2532	987	632	632
query94	702	418	332	332
query95	424	336	335	335
query96	485	582	280	280
query97	2903	2992	2872	2872
query98	251	217	217	217
query99	1332	1403	1275	1275
Total cold run time: 274325 ms
Total hot run time: 188487 ms

doris-robot · 2025-09-11T03:04:34Z

ClickBench: Total hot run time: 29.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 817eb80eea0f4a85ad8f7a44191a60f014e31240, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.60	0.11	0.11
query5	0.28	0.26	0.25
query6	1.18	0.66	0.64
query7	0.04	0.03	0.03
query8	0.05	0.05	0.05
query9	0.62	0.54	0.52
query10	0.59	0.57	0.57
query11	0.18	0.11	0.11
query12	0.15	0.13	0.12
query13	0.66	0.63	0.62
query14	1.02	1.06	1.02
query15	0.86	0.85	0.88
query16	0.41	0.39	0.39
query17	1.06	1.04	1.04
query18	0.22	0.20	0.20
query19	1.90	1.81	1.86
query20	0.02	0.01	0.02
query21	15.42	0.96	0.59
query22	0.76	1.07	0.73
query23	14.97	1.37	0.65
query24	6.69	1.72	0.69
query25	0.52	0.25	0.08
query26	0.60	0.18	0.14
query27	0.06	0.05	0.05
query28	9.91	0.94	0.42
query29	12.56	3.94	3.25
query30	0.32	0.13	0.11
query31	2.83	0.60	0.38
query32	3.25	0.56	0.48
query33	3.09	3.12	3.12
query34	16.27	5.49	4.93
query35	4.95	4.92	4.94
query36	0.69	0.52	0.50
query37	0.10	0.07	0.08
query38	0.06	0.05	0.05
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.67 s
Total hot run time: 29.94 s

doris-robot · 2025-09-11T03:21:53Z

BE UT Coverage Report

Increment line coverage 68.42% (13/19) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.97% (17304/33297)
Line Coverage	37.31% (157704/422657)
Region Coverage	31.96% (120363/376635)
Branch Coverage	33.33% (52830/158509)

hello-stephen · 2025-09-11T04:47:00Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (19/19) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.76% (23137/32696)
Line Coverage	57.15% (241385/422403)
Region Coverage	52.59% (200911/382064)
Branch Coverage	54.17% (86361/159412)

dataroaring

LGTM

github-actions · 2025-09-15T02:18:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-15T02:18:57Z

PR approved by anyone and no changes requested.

github-actions · 2025-09-15T02:21:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-15T02:21:33Z

PR approved by anyone and no changes requested.

liaoxin01

LGTM

kaijchen requested review from dataroaring and gavinchou as code owners July 8, 2025 12:42

kaijchen changed the title ~~[opt](load) use standalone thread pool for asynchronous delete_bitmap calculation~~ [opt](load) limit concurrency of delete_bitmap computation Jul 8, 2025

kaijchen force-pushed the DORIS-19726-1 branch from f2541db to 24a8499 Compare July 8, 2025 13:16

liaoxin01 requested a review from zhannngchen July 10, 2025 01:37

dataroaring force-pushed the DORIS-19726-1 branch from 2f3c87c to 78feb2a Compare September 8, 2025 08:55

kaijchen added 2 commits September 10, 2025 19:22

[opt](load) use standalone thread pool for asynchronous delete_bitmap…

9c13184

… calculation

rename

cedfd44

kaijchen force-pushed the DORIS-19726-1 branch from 78feb2a to 382ba20 Compare September 10, 2025 11:27

auto thread num = cpu core / 2

13a0b35

kaijchen force-pushed the DORIS-19726-1 branch from 382ba20 to 13a0b35 Compare September 10, 2025 13:05

fix style

817eb80

dataroaring approved these changes Sep 15, 2025

View reviewed changes

github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 15, 2025

dataroaring added dev/4.0.x and removed approved Indicates a PR has been approved by one committer. reviewed labels Sep 15, 2025

bobhan1 approved these changes Sep 15, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Sep 15, 2025

github-actions Bot added the reviewed label Sep 15, 2025

liaoxin01 approved these changes Sep 15, 2025

View reviewed changes

liaoxin01 merged commit ea0e95f into apache:master Sep 19, 2025
29 of 32 checks passed

yiguolei removed the dev/4.0.x label Sep 29, 2025

Conversation

kaijchen commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 8, 2025

Uh oh!

kaijchen commented Jul 8, 2025

Uh oh!

kaijchen commented Jul 8, 2025

Uh oh!

kaijchen commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

BE UT Coverage Report

Uh oh!

kaijchen commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring commented Sep 8, 2025

Uh oh!

doris-robot commented Sep 8, 2025

Uh oh!

doris-robot commented Sep 8, 2025

Uh oh!

doris-robot commented Sep 8, 2025

Uh oh!

kaijchen commented Sep 10, 2025

Uh oh!

kaijchen commented Sep 10, 2025

Uh oh!

doris-robot commented Sep 10, 2025

Uh oh!

doris-robot commented Sep 10, 2025

Uh oh!

doris-robot commented Sep 10, 2025

Uh oh!

hello-stephen commented Sep 10, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 10, 2025

BE Regression && UT Coverage Report

Uh oh!

kaijchen commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 11, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

kaijchen commented Jul 8, 2025 •

edited

Loading