Skip to content

[fix] (index compaction) Fix inverted index file size#37564

Merged
qidaye merged 5 commits intoapache:masterfrom
csun5285:fix_index_compaction_idx_size
Jul 15, 2024
Merged

[fix] (index compaction) Fix inverted index file size#37564
qidaye merged 5 commits intoapache:masterfrom
csun5285:fix_index_compaction_idx_size

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented Jul 9, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@csun5285 csun5285 force-pushed the fix_index_compaction_idx_size branch from f566a9b to ed571a0 Compare July 9, 2024 13:57
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

@csun5285 csun5285 force-pushed the fix_index_compaction_idx_size branch from ed571a0 to ebccf52 Compare July 9, 2024 14:01
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

1 similar comment
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 9, 2024

TPC-H: Total hot run time: 39499 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5e6562b1599e333e24a7d594ded54f1abdae566, data reload: false

------ Round 1 ----------------------------------
q1	17623	4307	4296	4296
q2	2023	196	196	196
q3	10455	1236	1066	1066
q4	10182	859	834	834
q5	7499	2647	2605	2605
q6	220	136	134	134
q7	942	590	590	590
q8	9245	2051	2077	2051
q9	8919	6460	6445	6445
q10	8994	3727	3737	3727
q11	452	235	236	235
q12	429	228	228	228
q13	18993	2953	2973	2953
q14	274	220	228	220
q15	531	484	501	484
q16	499	373	374	373
q17	969	679	667	667
q18	7834	7399	7271	7271
q19	3717	1482	1383	1383
q20	660	319	323	319
q21	4843	3087	3877	3087
q22	400	335	335	335
Total cold run time: 115703 ms
Total hot run time: 39499 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4267	4192	4192
q2	370	252	260	252
q3	2977	2766	2874	2766
q4	2045	1725	1704	1704
q5	5737	5492	5470	5470
q6	221	134	135	134
q7	2201	1952	1847	1847
q8	3256	3383	3405	3383
q9	8688	8726	8794	8726
q10	4030	3863	3710	3710
q11	617	509	485	485
q12	822	651	675	651
q13	15975	3196	3208	3196
q14	303	289	278	278
q15	536	492	468	468
q16	472	433	425	425
q17	1801	1546	1487	1487
q18	8072	7921	7842	7842
q19	2335	1685	1548	1548
q20	2157	1857	1841	1841
q21	5013	4875	4655	4655
q22	611	556	546	546
Total cold run time: 72605 ms
Total hot run time: 55606 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 9, 2024

TPC-DS: Total hot run time: 172965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5e6562b1599e333e24a7d594ded54f1abdae566, data reload: false

query1	915	372	362	362
query2	6445	2542	2477	2477
query3	6636	205	215	205
query4	28346	17337	17063	17063
query5	3641	490	472	472
query6	272	174	163	163
query7	4576	286	288	286
query8	321	306	297	297
query9	8846	2346	2341	2341
query10	423	283	274	274
query11	10492	9904	9895	9895
query12	135	81	79	79
query13	1635	362	361	361
query14	10116	8231	7101	7101
query15	235	185	188	185
query16	7184	312	306	306
query17	1796	543	520	520
query18	1190	298	267	267
query19	205	148	157	148
query20	93	78	80	78
query21	210	128	119	119
query22	4333	4249	4123	4123
query23	33746	33694	33438	33438
query24	10980	2785	2938	2785
query25	610	402	382	382
query26	977	150	144	144
query27	2282	268	277	268
query28	6889	2087	2088	2087
query29	881	628	644	628
query30	258	153	148	148
query31	980	776	732	732
query32	94	51	57	51
query33	755	315	286	286
query34	1023	501	488	488
query35	703	598	594	594
query36	1129	988	977	977
query37	146	84	89	84
query38	2914	2913	2794	2794
query39	895	854	806	806
query40	197	116	117	116
query41	51	54	51	51
query42	115	101	101	101
query43	590	582	548	548
query44	1206	720	761	720
query45	195	169	163	163
query46	1072	720	727	720
query47	1857	1753	1768	1753
query48	361	294	294	294
query49	844	407	417	407
query50	767	386	391	386
query51	7001	6699	6769	6699
query52	103	92	91	91
query53	357	286	283	283
query54	946	440	436	436
query55	73	71	72	71
query56	275	281	282	281
query57	1104	1048	1056	1048
query58	260	275	248	248
query59	3565	3379	3060	3060
query60	299	266	273	266
query61	98	91	98	91
query62	790	643	641	641
query63	325	289	280	280
query64	9321	2159	1628	1628
query65	3120	3134	3100	3100
query66	753	340	320	320
query67	15490	15153	15320	15153
query68	4473	523	515	515
query69	528	415	339	339
query70	1206	1154	1163	1154
query71	391	272	267	267
query72	7149	5448	5300	5300
query73	729	323	320	320
query74	5943	5496	5498	5496
query75	3402	2647	2672	2647
query76	2621	988	884	884
query77	468	311	290	290
query78	9691	8806	9193	8806
query79	2277	519	513	513
query80	1821	483	482	482
query81	603	220	226	220
query82	815	223	135	135
query83	248	162	162	162
query84	238	86	90	86
query85	1728	301	296	296
query86	487	340	292	292
query87	3244	3130	3091	3091
query88	4418	2422	2454	2422
query89	484	387	388	387
query90	1780	190	189	189
query91	131	103	103	103
query92	66	47	49	47
query93	2659	501	490	490
query94	1149	214	215	214
query95	466	311	317	311
query96	584	271	263	263
query97	3213	3018	3031	3018
query98	218	198	192	192
query99	1569	1281	1265	1265
Total cold run time: 279241 ms
Total hot run time: 172965 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 9, 2024

ClickBench: Total hot run time: 31.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5e6562b1599e333e24a7d594ded54f1abdae566, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.49	0.49
query6	1.14	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.48	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.11	0.12
query13	0.59	0.58	0.59
query14	0.76	0.79	0.77
query15	0.85	0.84	0.81
query16	0.36	0.37	0.34
query17	0.94	0.98	0.99
query18	0.25	0.22	0.22
query19	1.76	1.73	1.71
query20	0.01	0.01	0.01
query21	15.39	0.75	0.66
query22	3.76	7.26	2.55
query23	18.28	1.35	1.28
query24	2.15	0.25	0.23
query25	0.16	0.09	0.08
query26	0.28	0.20	0.20
query27	0.45	0.23	0.23
query28	13.22	1.02	0.98
query29	12.60	3.29	3.31
query30	0.26	0.06	0.05
query31	2.89	0.37	0.38
query32	3.26	0.48	0.47
query33	2.87	2.95	2.94
query34	17.11	4.32	4.29
query35	4.41	4.36	4.41
query36	0.65	0.46	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.13	0.11
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.1 s
Total hot run time: 31.12 s

@csun5285 csun5285 force-pushed the fix_index_compaction_idx_size branch from f5e6562 to f17c92f Compare July 10, 2024 05:42
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 10, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 39984 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f17c92fad32e398ae39ddbed35a0d1c73a77211e, data reload: false

------ Round 1 ----------------------------------
q1	17624	4567	4292	4292
q2	2029	210	197	197
q3	10422	1235	1015	1015
q4	10189	801	711	711
q5	7503	2704	2656	2656
q6	225	142	139	139
q7	968	602	600	600
q8	9238	2116	2087	2087
q9	9008	6532	6516	6516
q10	8993	3751	3749	3749
q11	478	243	238	238
q12	406	236	227	227
q13	17913	3024	2999	2999
q14	271	222	237	222
q15	522	481	496	481
q16	524	380	380	380
q17	1000	579	656	579
q18	8126	7535	7427	7427
q19	3938	1630	1506	1506
q20	683	328	326	326
q21	5120	3293	3872	3293
q22	400	344	356	344
Total cold run time: 115580 ms
Total hot run time: 39984 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4361	4199	4205	4199
q2	374	282	259	259
q3	3020	2823	2912	2823
q4	2025	1687	1673	1673
q5	5665	5561	5536	5536
q6	224	134	136	134
q7	2191	1906	1929	1906
q8	3358	3506	3478	3478
q9	8842	8756	8805	8756
q10	4132	3780	3840	3780
q11	613	510	488	488
q12	847	647	639	639
q13	15781	3221	3233	3221
q14	317	280	267	267
q15	551	476	510	476
q16	475	447	431	431
q17	1834	1554	1524	1524
q18	8232	7860	7830	7830
q19	2391	1627	1688	1627
q20	2095	1868	1860	1860
q21	5314	4869	4743	4743
q22	681	566	576	566
Total cold run time: 73323 ms
Total hot run time: 56216 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 174409 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f17c92fad32e398ae39ddbed35a0d1c73a77211e, data reload: false

query1	926	373	381	373
query2	6436	2474	2434	2434
query3	6636	208	211	208
query4	28183	17556	17305	17305
query5	3669	480	490	480
query6	259	168	158	158
query7	4590	295	291	291
query8	322	305	300	300
query9	8500	2403	2384	2384
query10	435	292	294	292
query11	10934	10015	10068	10015
query12	119	87	82	82
query13	1657	378	378	378
query14	9978	7866	7380	7380
query15	257	189	187	187
query16	7785	316	309	309
query17	1812	564	533	533
query18	1959	277	280	277
query19	196	148	153	148
query20	90	82	81	81
query21	203	133	133	133
query22	4228	4016	3941	3941
query23	34244	33897	33350	33350
query24	11891	2912	2900	2900
query25	616	393	398	393
query26	1617	157	150	150
query27	2889	283	291	283
query28	7708	2133	2105	2105
query29	1033	637	638	637
query30	247	161	151	151
query31	938	790	762	762
query32	94	51	57	51
query33	744	299	323	299
query34	985	514	523	514
query35	697	591	595	591
query36	1140	989	951	951
query37	146	88	80	80
query38	2980	2874	2782	2782
query39	865	800	805	800
query40	247	127	121	121
query41	53	53	52	52
query42	126	100	106	100
query43	613	575	546	546
query44	1207	753	745	745
query45	199	158	163	158
query46	1084	712	687	687
query47	1858	1749	1744	1744
query48	384	298	314	298
query49	893	412	427	412
query50	784	401	400	400
query51	6866	6809	6767	6767
query52	120	93	98	93
query53	362	300	295	295
query54	962	450	465	450
query55	76	71	74	71
query56	293	264	268	264
query57	1156	1050	1055	1050
query58	265	245	255	245
query59	3430	3090	3215	3090
query60	304	272	279	272
query61	97	100	93	93
query62	797	661	641	641
query63	317	289	301	289
query64	10323	2240	7514	2240
query65	3314	3103	3113	3103
query66	1247	339	331	331
query67	15554	15166	15060	15060
query68	4511	556	539	539
query69	474	327	323	323
query70	1189	1126	1157	1126
query71	376	288	283	283
query72	6856	5112	5538	5112
query73	730	332	326	326
query74	5897	5500	5523	5500
query75	3383	2677	2731	2677
query76	2655	1007	944	944
query77	431	311	306	306
query78	9592	8989	8889	8889
query79	2655	522	512	512
query80	2045	489	478	478
query81	590	223	217	217
query82	963	139	137	137
query83	323	173	163	163
query84	275	95	87	87
query85	1910	387	305	305
query86	471	318	307	307
query87	3294	3112	3131	3112
query88	4215	2452	2461	2452
query89	478	399	388	388
query90	1867	199	186	186
query91	133	107	104	104
query92	59	49	49	49
query93	4295	534	525	525
query94	1161	213	209	209
query95	408	321	313	313
query96	606	270	274	270
query97	3177	3033	3041	3033
query98	222	205	198	198
query99	1580	1234	1269	1234
Total cold run time: 287760 ms
Total hot run time: 174409 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f17c92fad32e398ae39ddbed35a0d1c73a77211e, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.03
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.54	0.53	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.61	0.58	0.58
query14	0.77	0.78	0.78
query15	0.86	0.82	0.82
query16	0.37	0.36	0.36
query17	1.01	1.00	1.00
query18	0.23	0.22	0.21
query19	1.84	1.74	1.74
query20	0.01	0.01	0.00
query21	15.38	0.79	0.67
query22	4.11	7.10	2.00
query23	18.29	1.40	1.24
query24	2.07	0.22	0.22
query25	0.16	0.11	0.09
query26	0.29	0.20	0.21
query27	0.45	0.23	0.23
query28	13.46	1.03	1.00
query29	12.59	3.32	3.28
query30	0.25	0.06	0.05
query31	2.88	0.40	0.39
query32	3.26	0.47	0.48
query33	2.90	2.96	2.89
query34	17.11	4.35	4.32
query35	4.46	4.38	4.42
query36	0.65	0.47	0.48
query37	0.19	0.16	0.16
query38	0.15	0.14	0.15
query39	0.05	0.03	0.04
query40	0.15	0.13	0.12
query41	0.08	0.04	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 30.64 s

airborne12
airborne12 reviewed Jul 10, 2024
View reviewed changes
Comment thread be/src/olap/compaction.cpp Outdated
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 10, 2024

run buildall

1 similar comment
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 10, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 39635 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52e3887f1fba20e20bc899c049424f023422769d, data reload: false

------ Round 1 ----------------------------------
q1	17612	4314	4325	4314
q2	2012	189	183	183
q3	10447	1216	998	998
q4	10189	743	832	743
q5	7488	2667	2737	2667
q6	221	136	136	136
q7	952	610	602	602
q8	9227	2073	2048	2048
q9	9101	6487	6487	6487
q10	9015	3725	3719	3719
q11	450	241	244	241
q12	467	231	242	231
q13	18872	2981	2929	2929
q14	256	226	221	221
q15	528	488	475	475
q16	495	372	368	368
q17	974	630	525	525
q18	8077	7569	7454	7454
q19	7396	1476	1459	1459
q20	684	326	325	325
q21	4876	3181	3924	3181
q22	379	329	334	329
Total cold run time: 119718 ms
Total hot run time: 39635 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4401	4305	4292	4292
q2	366	273	273	273
q3	3065	2828	2918	2828
q4	1959	1759	1725	1725
q5	5547	5500	5425	5425
q6	228	137	131	131
q7	2238	1882	1862	1862
q8	3267	3457	3415	3415
q9	8682	8795	8689	8689
q10	4129	3701	3882	3701
q11	596	519	495	495
q12	841	656	657	656
q13	17271	3176	3169	3169
q14	317	265	292	265
q15	536	476	478	476
q16	482	420	422	420
q17	1806	1512	1532	1512
q18	8174	8043	7784	7784
q19	1774	1625	1625	1625
q20	2254	1855	1854	1854
q21	7567	4760	5054	4760
q22	654	564	560	560
Total cold run time: 76154 ms
Total hot run time: 55917 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 173339 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52e3887f1fba20e20bc899c049424f023422769d, data reload: false

query1	927	374	369	369
query2	6405	2383	2325	2325
query3	6639	203	222	203
query4	28592	17252	17365	17252
query5	3776	502	464	464
query6	255	164	165	164
query7	4588	290	287	287
query8	341	291	288	288
query9	8734	2383	2360	2360
query10	443	278	280	278
query11	12679	9916	9981	9916
query12	115	82	85	82
query13	1700	388	378	378
query14	10079	7776	7534	7534
query15	227	179	181	179
query16	7623	310	297	297
query17	1673	537	520	520
query18	1886	288	274	274
query19	196	150	147	147
query20	92	80	81	80
query21	199	128	122	122
query22	4268	4081	3947	3947
query23	34050	33430	35300	33430
query24	10718	2843	2926	2843
query25	536	375	394	375
query26	700	152	153	152
query27	2264	289	289	289
query28	5951	2156	2138	2138
query29	870	625	628	625
query30	245	167	153	153
query31	962	728	762	728
query32	107	56	54	54
query33	668	308	284	284
query34	914	489	497	489
query35	704	574	569	569
query36	1115	968	973	968
query37	146	85	82	82
query38	2948	2834	2878	2834
query39	886	806	796	796
query40	204	120	116	116
query41	57	51	52	51
query42	118	100	109	100
query43	581	563	542	542
query44	1093	737	746	737
query45	191	164	159	159
query46	1081	729	717	717
query47	1856	1792	1734	1734
query48	372	299	304	299
query49	851	435	421	421
query50	818	393	389	389
query51	6838	6783	6667	6667
query52	103	90	90	90
query53	355	290	297	290
query54	844	448	446	446
query55	75	72	72	72
query56	280	263	260	260
query57	1091	1051	1017	1017
query58	243	240	241	240
query59	3569	3250	3310	3250
query60	304	285	273	273
query61	94	94	94	94
query62	795	665	649	649
query63	325	284	291	284
query64	9245	2213	1643	1643
query65	3192	3168	3170	3168
query66	701	336	327	327
query67	15515	15052	14854	14854
query68	4519	548	546	546
query69	552	358	338	338
query70	1124	1148	1188	1148
query71	368	268	276	268
query72	7471	5727	5204	5204
query73	742	326	327	326
query74	5927	5412	5522	5412
query75	3402	2642	2690	2642
query76	2284	931	856	856
query77	509	315	312	312
query78	9651	8907	8868	8868
query79	2221	535	522	522
query80	2329	474	456	456
query81	600	216	214	214
query82	866	135	133	133
query83	300	170	164	164
query84	264	91	83	83
query85	2046	312	296	296
query86	482	319	334	319
query87	3264	3092	3090	3090
query88	4284	2471	2471	2471
query89	482	399	389	389
query90	1672	196	193	193
query91	128	105	101	101
query92	62	52	50	50
query93	2522	508	504	504
query94	918	268	211	211
query95	392	309	308	308
query96	609	285	274	274
query97	3194	3029	3019	3019
query98	221	214	191	191
query99	1664	1286	1267	1267
Total cold run time: 281081 ms
Total hot run time: 173339 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52e3887f1fba20e20bc899c049424f023422769d, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.67	0.08	0.10
query5	0.50	0.48	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.55	0.53	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.78	0.78
query15	0.85	0.82	0.79
query16	0.36	0.36	0.36
query17	0.95	1.01	1.06
query18	0.23	0.22	0.22
query19	1.79	1.72	1.74
query20	0.02	0.01	0.00
query21	15.87	0.76	0.67
query22	3.74	7.63	2.38
query23	18.27	1.38	1.35
query24	2.17	0.22	0.22
query25	0.17	0.09	0.08
query26	0.30	0.20	0.21
query27	0.46	0.24	0.22
query28	13.32	1.02	1.01
query29	12.64	3.32	3.31
query30	0.25	0.06	0.05
query31	2.88	0.38	0.39
query32	3.28	0.49	0.47
query33	2.96	2.88	2.88
query34	17.14	4.43	4.38
query35	4.44	4.45	4.46
query36	0.65	0.48	0.48
query37	0.18	0.16	0.16
query38	0.16	0.16	0.14
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.95 s
Total hot run time: 31.27 s

xiaokang
xiaokang reviewed Jul 10, 2024
View reviewed changes
Comment thread be/src/olap/compaction.cpp Outdated
@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 11, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

TPC-H: Total hot run time: 40039 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8c7c8a724f8952e53212b62db91e34b05ed56ee, data reload: false

------ Round 1 ----------------------------------
q1	17604	4497	4293	4293
q2	2022	192	184	184
q3	10467	1189	1124	1124
q4	10196	724	870	724
q5	7528	2660	2660	2660
q6	220	137	141	137
q7	963	601	594	594
q8	9214	2057	2063	2057
q9	8758	6552	6546	6546
q10	8932	3796	3784	3784
q11	448	239	247	239
q12	443	223	227	223
q13	18810	2982	3003	2982
q14	278	232	247	232
q15	515	486	473	473
q16	504	387	384	384
q17	961	687	703	687
q18	8032	7499	7451	7451
q19	6060	1552	1536	1536
q20	690	314	327	314
q21	4875	3079	3247	3079
q22	385	336	337	336
Total cold run time: 117905 ms
Total hot run time: 40039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4220	4210	4210
q2	372	269	279	269
q3	2999	2900	2945	2900
q4	2040	1669	1760	1669
q5	5601	5544	5503	5503
q6	220	137	140	137
q7	2241	1877	1838	1838
q8	3231	3422	3439	3422
q9	8744	8850	8816	8816
q10	4146	3749	3858	3749
q11	595	512	504	504
q12	850	648	621	621
q13	17166	3168	3235	3168
q14	326	288	295	288
q15	530	497	494	494
q16	510	450	436	436
q17	1829	1504	1504	1504
q18	8260	8092	7927	7927
q19	1784	1602	1674	1602
q20	2158	1908	1864	1864
q21	5743	4780	4935	4780
q22	668	581	578	578
Total cold run time: 74368 ms
Total hot run time: 56279 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 12, 2024

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e67d1512f1710fd7f9a7bcff60f74e4f327c3c2, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.48	0.52
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.49	0.48
query10	0.56	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.13
query13	0.59	0.58	0.58
query14	0.75	0.78	0.77
query15	0.85	0.81	0.81
query16	0.35	0.37	0.36
query17	0.99	1.02	1.02
query18	0.23	0.21	0.22
query19	1.82	1.82	1.69
query20	0.01	0.01	0.02
query21	15.39	0.74	0.65
query22	4.58	7.09	2.07
query23	18.42	1.43	1.24
query24	2.20	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.22	0.20
query27	0.46	0.23	0.24
query28	13.22	1.03	1.00
query29	12.64	3.27	3.29
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.48	0.48
query33	2.88	2.89	2.92
query34	17.02	4.37	4.31
query35	4.46	4.42	4.40
query36	0.64	0.46	0.48
query37	0.18	0.15	0.14
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.03
Total cold run time: 110.19 s
Total hot run time: 30.63 s

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 12, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 12, 2024

TPC-H: Total hot run time: 39659 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5377192d77cda09c76bb237366c1dd45bb9defc, data reload: false

------ Round 1 ----------------------------------
q1	17636	4373	4298	4298
q2	2016	194	186	186
q3	10454	1236	1094	1094
q4	10196	817	868	817
q5	7526	2676	2636	2636
q6	222	136	141	136
q7	961	609	602	602
q8	9213	2064	2102	2064
q9	8688	6509	6530	6509
q10	8823	3802	3750	3750
q11	458	238	241	238
q12	428	230	232	230
q13	17762	2995	2967	2967
q14	280	235	242	235
q15	524	486	502	486
q16	480	392	395	392
q17	962	613	733	613
q18	8060	7489	7359	7359
q19	7543	1454	1383	1383
q20	703	327	316	316
q21	4987	3070	3171	3070
q22	346	290	278	278
Total cold run time: 118268 ms
Total hot run time: 39659 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4397	4295	4292	4292
q2	369	282	258	258
q3	3026	2896	2921	2896
q4	1981	1723	1754	1723
q5	5614	5508	5497	5497
q6	223	142	134	134
q7	2245	1889	1776	1776
q8	3293	3410	3419	3410
q9	8780	8808	8897	8808
q10	4158	3847	3855	3847
q11	586	496	511	496
q12	817	633	635	633
q13	16303	3220	3178	3178
q14	312	287	287	287
q15	532	478	488	478
q16	483	447	448	447
q17	1787	1496	1466	1466
q18	8174	7987	7760	7760
q19	1761	1739	1569	1569
q20	2173	1905	1872	1872
q21	7404	4881	4720	4720
q22	642	519	530	519
Total cold run time: 75060 ms
Total hot run time: 56066 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 12, 2024

TPC-DS: Total hot run time: 172785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5377192d77cda09c76bb237366c1dd45bb9defc, data reload: false

query1	928	374	369	369
query2	6438	1895	1888	1888
query3	6639	209	215	209
query4	28296	17668	17512	17512
query5	3671	497	479	479
query6	254	163	171	163
query7	4578	304	289	289
query8	241	217	200	200
query9	8446	2392	2382	2382
query10	434	288	270	270
query11	11653	10178	10086	10086
query12	130	91	85	85
query13	1634	384	372	372
query14	10624	6901	7865	6901
query15	230	168	165	165
query16	7395	329	312	312
query17	1352	549	525	525
query18	1854	279	272	272
query19	205	151	163	151
query20	94	84	87	84
query21	201	134	123	123
query22	4257	4010	4079	4010
query23	34069	33550	33574	33550
query24	11209	3006	2874	2874
query25	631	415	402	402
query26	1047	155	149	149
query27	2350	282	278	278
query28	6765	2093	2083	2083
query29	903	636	645	636
query30	242	153	152	152
query31	1002	766	744	744
query32	97	56	56	56
query33	757	302	293	293
query34	1023	504	509	504
query35	681	594	602	594
query36	1115	969	991	969
query37	153	88	89	88
query38	2957	2918	2803	2803
query39	890	846	844	844
query40	206	127	126	126
query41	50	45	45	45
query42	118	99	100	99
query43	490	469	476	469
query44	1224	746	744	744
query45	198	162	161	161
query46	1089	739	714	714
query47	1836	1761	1769	1761
query48	378	292	313	292
query49	839	411	416	411
query50	777	393	393	393
query51	6932	6859	6834	6834
query52	107	104	93	93
query53	395	305	297	297
query54	888	459	459	459
query55	77	71	72	71
query56	300	269	275	269
query57	1137	1051	1029	1029
query58	244	271	264	264
query59	2785	2634	2586	2586
query60	302	282	286	282
query61	101	93	93	93
query62	803	648	647	647
query63	324	296	291	291
query64	9405	2248	1680	1680
query65	3165	3102	3119	3102
query66	744	327	320	320
query67	15402	14954	14902	14902
query68	4584	563	539	539
query69	636	411	334	334
query70	1163	1138	1135	1135
query71	379	277	273	273
query72	7453	5497	5077	5077
query73	762	328	331	328
query74	5982	5552	5495	5495
query75	3364	2748	2704	2704
query76	2119	944	940	940
query77	621	304	302	302
query78	9640	8992	8976	8976
query79	2150	534	544	534
query80	2103	486	477	477
query81	598	221	223	221
query82	984	138	134	134
query83	291	172	173	172
query84	263	88	93	88
query85	1145	319	297	297
query86	450	350	337	337
query87	3285	3082	3123	3082
query88	3739	2500	2469	2469
query89	478	406	392	392
query90	1670	199	197	197
query91	132	106	100	100
query92	60	51	53	51
query93	2066	521	523	521
query94	1076	215	219	215
query95	417	323	330	323
query96	610	287	285	285
query97	3204	3041	3101	3041
query98	227	212	190	190
query99	1528	1274	1256	1256
Total cold run time: 278940 ms
Total hot run time: 172785 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 12, 2024

ClickBench: Total hot run time: 30.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5377192d77cda09c76bb237366c1dd45bb9defc, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.04	0.04
query4	1.67	0.08	0.08
query5	0.51	0.49	0.48
query6	1.16	0.71	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.54	0.53
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.59	0.59	0.60
query14	0.79	0.76	0.78
query15	0.84	0.81	0.82
query16	0.38	0.36	0.37
query17	0.99	1.01	0.97
query18	0.23	0.22	0.23
query19	1.90	1.73	1.75
query20	0.02	0.01	0.01
query21	15.38	0.74	0.66
query22	3.51	8.17	2.12
query23	18.30	1.45	1.26
query24	2.07	0.22	0.22
query25	0.14	0.08	0.08
query26	0.31	0.21	0.21
query27	0.46	0.24	0.24
query28	13.36	1.03	1.02
query29	12.64	3.46	3.37
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.26	0.49	0.48
query33	2.86	2.95	2.89
query34	17.07	4.37	4.36
query35	4.45	4.43	4.51
query36	0.65	0.46	0.50
query37	0.19	0.17	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.08	0.04	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.05
Total cold run time: 109.2 s
Total hot run time: 30.98 s

airborne12
airborne12 approved these changes Jul 15, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2024

PR approved by at least one committer and no changes requested.

@qidaye qidaye merged commit 68b8927 into apache:master Jul 15, 2024
This was referenced Jul 15, 2024
Merged
Merged
airborne12 pushed a commit that referenced this pull request Jul 16, 2024
@csun5285
[cherry-pick](branch-2.1) fix inverted index file size (#37836)
253f929 
## Proposed changes

pick from master #37232
pick from master #37564
@csun5285 csun5285 mentioned this pull request Jul 17, 2024
Closed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@csun5285 @seawinde
[fix](index compaction) Fix inverted index file size (apache#37564)
1e0f028
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[fix](index compaction) Fix inverted index file size (#37564)
d3a206c
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang left review comments

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @doris-robot @xiaokang @airborne12 @qidaye @dataroaring