Skip to content

[chore](table) Add batch method to get visible version of the olap table#38949

Merged
w41ter merged 1 commit intoapache:masterfrom
w41ter:batch_get_table_versions
Aug 12, 2024
Merged

[chore](table) Add batch method to get visible version of the olap table#38949
w41ter merged 1 commit intoapache:masterfrom
w41ter:batch_get_table_versions

Conversation

@w41ter
Copy link
Copy Markdown
Contributor

@w41ter w41ter commented Aug 6, 2024

Since get visible version is a heavy operation in the cloud mode, this PR add a batch method, to obtain all visible versions via only one RPC.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions Bot added the doing label Aug 6, 2024
@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 6, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 6, 2024

TPC-H: Total hot run time: 41935 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f9ec4e0f4896a4f3eb6cde8a12446c1941bc3d2, data reload: false

------ Round 1 ----------------------------------
q1	18250	4250	4191	4191
q2	3026	206	204	204
q3	11236	1281	1398	1281
q4	11098	832	965	832
q5	8252	2940	2998	2940
q6	221	140	141	140
q7	1066	642	623	623
q8	9465	1857	1935	1857
q9	8517	6595	6625	6595
q10	8735	3889	3862	3862
q11	424	255	260	255
q12	409	236	230	230
q13	17761	2970	2971	2970
q14	274	241	245	241
q15	546	490	482	482
q16	522	407	409	407
q17	987	952	938	938
q18	8104	7350	7281	7281
q19	1503	1235	1224	1224
q20	583	328	344	328
q21	5309	4772	4838	4772
q22	354	282	285	282
Total cold run time: 116642 ms
Total hot run time: 41935 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4090	4059	4048	4048
q2	330	224	226	224
q3	2988	3013	3018	3013
q4	1902	1906	1896	1896
q5	5274	5259	5248	5248
q6	217	129	131	129
q7	2048	1700	1694	1694
q8	3194	3266	3253	3253
q9	8278	8365	8353	8353
q10	3808	3868	3888	3868
q11	548	451	442	442
q12	751	587	550	550
q13	14429	2979	2964	2964
q14	285	254	257	254
q15	513	469	476	469
q16	439	399	397	397
q17	1743	1731	1705	1705
q18	7758	7347	7298	7298
q19	1676	1667	1691	1667
q20	1987	1761	1760	1760
q21	5588	5309	5242	5242
q22	535	453	470	453
Total cold run time: 68381 ms
Total hot run time: 54927 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 6, 2024

TPC-DS: Total hot run time: 168830 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f9ec4e0f4896a4f3eb6cde8a12446c1941bc3d2, data reload: false

query1	928	395	378	378
query2	6471	1710	1711	1710
query3	6663	214	226	214
query4	19524	17741	17266	17266
query5	4297	529	531	529
query6	283	177	186	177
query7	4618	297	297	297
query8	255	212	202	202
query9	8499	2385	2371	2371
query10	432	273	285	273
query11	10482	10132	10069	10069
query12	141	88	86	86
query13	1625	379	366	366
query14	9202	6960	7703	6960
query15	205	160	169	160
query16	7094	491	478	478
query17	939	551	544	544
query18	1861	280	294	280
query19	195	153	147	147
query20	94	86	85	85
query21	211	101	104	101
query22	4307	4015	3986	3986
query23	33953	32836	33057	32836
query24	10318	3137	3052	3052
query25	702	377	398	377
query26	1782	153	151	151
query27	2956	279	284	279
query28	6991	1985	1978	1978
query29	1321	419	414	414
query30	282	153	152	152
query31	923	757	766	757
query32	106	60	59	59
query33	709	336	324	324
query34	924	479	502	479
query35	854	729	730	729
query36	1010	854	885	854
query37	296	82	80	80
query38	2887	2738	2746	2738
query39	908	818	812	812
query40	288	117	120	117
query41	50	45	45	45
query42	125	104	108	104
query43	447	406	412	406
query44	1206	735	720	720
query45	202	180	176	176
query46	1104	809	778	778
query47	1810	1723	1727	1723
query48	372	291	287	287
query49	1193	440	436	436
query50	919	450	450	450
query51	6774	6768	6718	6718
query52	106	88	90	88
query53	257	193	192	192
query54	685	458	468	458
query55	79	75	76	75
query56	273	274	275	274
query57	1163	1038	1042	1038
query58	293	279	271	271
query59	2548	2174	2355	2174
query60	296	277	280	277
query61	101	92	97	92
query62	917	660	660	660
query63	220	192	188	188
query64	5895	1942	1886	1886
query65	3168	3104	3123	3104
query66	1425	343	336	336
query67	15421	14833	14874	14833
query68	4362	583	585	583
query69	441	313	307	307
query70	1079	1021	1045	1021
query71	375	290	292	290
query72	7133	2683	2499	2499
query73	774	343	334	334
query74	5962	5614	5687	5614
query75	3438	2743	2747	2743
query76	2251	1228	1289	1228
query77	425	317	334	317
query78	9384	8947	8835	8835
query79	1433	538	541	538
query80	1214	525	518	518
query81	570	222	230	222
query82	1112	134	130	130
query83	240	170	175	170
query84	275	82	81	81
query85	1323	326	303	303
query86	396	327	302	302
query87	3279	3120	3093	3093
query88	2945	2439	2514	2439
query89	393	298	297	297
query90	1769	193	197	193
query91	133	104	103	103
query92	64	51	52	51
query93	1393	638	621	621
query94	876	307	287	287
query95	384	277	282	277
query96	603	283	279	279
query97	3256	3056	3053	3053
query98	212	210	190	190
query99	1629	1278	1256	1256
Total cold run time: 261708 ms
Total hot run time: 168830 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 6, 2024

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f9ec4e0f4896a4f3eb6cde8a12446c1941bc3d2, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.03
query3	0.22	0.04	0.04
query4	1.69	0.07	0.07
query5	0.52	0.49	0.49
query6	1.14	0.72	0.70
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.59	0.52	0.54
query10	0.57	0.58	0.58
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.77	0.78	0.82
query15	0.94	0.88	0.87
query16	0.36	0.37	0.35
query17	1.03	0.99	1.01
query18	0.22	0.21	0.21
query19	1.88	1.79	1.81
query20	0.01	0.01	0.01
query21	15.40	0.75	0.65
query22	3.77	7.80	1.42
query23	18.14	1.34	1.31
query24	2.27	0.22	0.22
query25	0.20	0.08	0.08
query26	0.33	0.22	0.21
query27	0.46	0.23	0.23
query28	13.17	1.02	0.96
query29	12.51	3.26	3.21
query30	0.25	0.06	0.05
query31	2.87	0.41	0.40
query32	3.24	0.50	0.48
query33	2.96	2.98	2.99
query34	15.42	4.28	4.28
query35	4.32	4.29	4.31
query36	0.69	0.48	0.49
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.78 s
Total hot run time: 30.19 s

dataroaring
dataroaring previously approved these changes Aug 7, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Aug 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 7, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Aug 7, 2024

PR approved by anyone and no changes requested.

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 7, 2024

run buildall

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 8, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 8, 2024

TPC-H: Total hot run time: 38909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cfbbb9411677c7f5fcb6326827ea12b9de150974, data reload: false

------ Round 1 ----------------------------------
q1	17664	4374	4357	4357
q2	2019	180	184	180
q3	10522	1172	1043	1043
q4	10129	742	715	715
q5	7488	2528	2511	2511
q6	237	146	143	143
q7	971	597	577	577
q8	9218	1897	1895	1895
q9	8502	6521	6550	6521
q10	7061	2247	2165	2165
q11	451	244	236	236
q12	526	219	213	213
q13	17952	2973	2992	2973
q14	276	234	228	228
q15	521	496	488	488
q16	510	395	379	379
q17	963	634	731	634
q18	7951	7466	7304	7304
q19	4556	987	923	923
q20	672	320	315	315
q21	5303	4152	4070	4070
q22	1132	1039	1045	1039
Total cold run time: 114624 ms
Total hot run time: 38909 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4518	4309	4315	4309
q2	385	276	270	270
q3	2819	2621	2604	2604
q4	1991	1686	1683	1683
q5	5532	5617	5510	5510
q6	233	146	140	140
q7	2155	1703	1716	1703
q8	3353	3484	3412	3412
q9	8742	8712	8752	8712
q10	3552	3228	3274	3228
q11	590	510	511	510
q12	817	611	618	611
q13	15779	3056	3178	3056
q14	298	296	281	281
q15	542	505	502	502
q16	481	436	435	435
q17	1813	1545	1521	1521
q18	8047	7982	7720	7720
q19	1755	1615	1580	1580
q20	2154	1903	1925	1903
q21	10733	5400	5399	5399
q22	1107	1022	1015	1015
Total cold run time: 77396 ms
Total hot run time: 56104 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 8, 2024

TPC-DS: Total hot run time: 203119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cfbbb9411677c7f5fcb6326827ea12b9de150974, data reload: false

query1	943	407	404	404
query2	6433	2053	1972	1972
query3	6628	206	211	206
query4	34262	23161	23052	23052
query5	3622	496	484	484
query6	268	170	164	164
query7	4562	292	284	284
query8	231	207	195	195
query9	8571	2377	2351	2351
query10	940	891	872	872
query11	16756	14844	14853	14844
query12	131	100	95	95
query13	1651	400	380	380
query14	9526	8123	7968	7968
query15	381	329	315	315
query16	7263	512	492	492
query17	1579	570	559	559
query18	1464	404	410	404
query19	254	215	198	198
query20	118	112	115	112
query21	211	106	104	104
query22	4512	4351	4217	4217
query23	34138	34183	33597	33597
query24	10749	3001	2935	2935
query25	582	386	379	379
query26	708	160	153	153
query27	2172	282	281	281
query28	5951	2020	1998	1998
query29	691	406	405	405
query30	254	172	145	145
query31	949	759	741	741
query32	98	57	55	55
query33	620	275	284	275
query34	868	464	470	464
query35	954	865	892	865
query36	1034	885	884	884
query37	129	82	78	78
query38	4192	4053	4184	4053
query39	1412	1367	1352	1352
query40	199	118	113	113
query41	46	44	43	43
query42	117	99	97	97
query43	526	463	476	463
query44	1069	726	727	726
query45	414	385	376	376
query46	1131	809	814	809
query47	1808	1734	1785	1734
query48	375	303	297	297
query49	837	410	426	410
query50	805	406	402	402
query51	6797	6704	6571	6571
query52	99	86	86	86
query53	246	183	178	178
query54	892	437	446	437
query55	76	74	74	74
query56	267	246	242	242
query57	1116	1048	1053	1048
query58	228	225	220	220
query59	3014	2970	2638	2638
query60	284	257	245	245
query61	130	96	95	95
query62	787	644	653	644
query63	211	177	174	174
query64	9234	2416	1900	1900
query65	3195	3123	3165	3123
query66	691	327	327	327
query67	15287	15410	14831	14831
query68	4493	588	534	534
query69	405	371	397	371
query70	1144	1089	1108	1089
query71	401	275	263	263
query72	17391	16487	16684	16487
query73	756	322	322	322
query74	9052	8753	8742	8742
query75	3359	2657	2691	2657
query76	2644	1004	1028	1004
query77	450	309	317	309
query78	9632	9135	9052	9052
query79	2685	522	514	514
query80	2435	482	481	481
query81	611	224	225	224
query82	802	149	132	132
query83	311	166	151	151
query84	266	80	79	79
query85	1855	311	369	311
query86	477	299	286	286
query87	4658	4528	4477	4477
query88	4304	2486	2488	2486
query89	399	283	282	282
query90	1839	196	199	196
query91	145	117	124	117
query92	63	50	50	50
query93	1996	528	527	527
query94	778	296	299	296
query95	351	261	264	261
query96	596	278	273	273
query97	3228	3062	3059	3059
query98	218	207	246	207
query99	1732	1265	1245	1245
Total cold run time: 305164 ms
Total hot run time: 203119 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 8, 2024

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cfbbb9411677c7f5fcb6326827ea12b9de150974, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.47
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.76	0.82	0.78
query15	0.85	0.82	0.82
query16	0.38	0.37	0.37
query17	1.04	1.04	0.95
query18	0.24	0.22	0.22
query19	1.78	1.71	1.72
query20	0.02	0.01	0.01
query21	15.44	0.72	0.65
query22	4.34	6.44	2.49
query23	18.29	1.37	1.35
query24	2.16	0.22	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.22	0.23
query28	13.23	1.03	1.01
query29	12.66	3.31	3.27
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.28	0.49	0.47
query33	2.83	2.94	2.88
query34	17.04	4.35	4.36
query35	4.39	4.40	4.43
query36	0.66	0.46	0.48
query37	0.19	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.06	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.05
Total cold run time: 109.83 s
Total hot run time: 31.21 s

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 9, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

TPC-H: Total hot run time: 39464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d00c72196ea02ddb2c9bd2b4839be052d553b98f, data reload: false

------ Round 1 ----------------------------------
q1	17619	4415	4289	4289
q2	2019	177	177	177
q3	10489	1166	1122	1122
q4	10137	718	725	718
q5	7495	2542	2493	2493
q6	225	140	140	140
q7	970	590	601	590
q8	9218	1897	1926	1897
q9	8880	6555	6601	6555
q10	7051	2177	2206	2177
q11	461	243	248	243
q12	419	221	222	221
q13	19140	2971	2962	2962
q14	279	238	233	233
q15	517	484	486	484
q16	490	379	379	379
q17	959	664	687	664
q18	7945	7405	7456	7405
q19	5502	970	1054	970
q20	688	350	341	341
q21	5329	4381	4641	4381
q22	1137	1031	1023	1023
Total cold run time: 116969 ms
Total hot run time: 39464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4268	4233	4233
q2	373	268	268	268
q3	2862	2646	2820	2646
q4	1997	1772	1719	1719
q5	5655	5512	5533	5512
q6	217	129	131	129
q7	2103	1778	1805	1778
q8	3289	3436	3462	3436
q9	8720	8801	8838	8801
q10	3552	3401	3169	3169
q11	621	521	498	498
q12	842	694	660	660
q13	15966	3178	3176	3176
q14	319	294	281	281
q15	545	503	481	481
q16	486	438	445	438
q17	1805	1517	1531	1517
q18	8089	8010	7808	7808
q19	1712	1572	1575	1572
q20	2104	1950	1887	1887
q21	7679	5507	5200	5200
q22	1165	1095	1048	1048
Total cold run time: 74499 ms
Total hot run time: 56257 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

TPC-DS: Total hot run time: 201558 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d00c72196ea02ddb2c9bd2b4839be052d553b98f, data reload: false

query1	906	360	355	355
query2	6438	1882	1824	1824
query3	6628	203	221	203
query4	34166	23297	22951	22951
query5	3600	480	492	480
query6	264	175	173	173
query7	4573	292	288	288
query8	262	217	208	208
query9	8441	2488	2442	2442
query10	521	470	448	448
query11	17747	14875	15018	14875
query12	139	103	95	95
query13	1649	380	373	373
query14	10307	8482	7642	7642
query15	311	222	211	211
query16	7666	520	492	492
query17	1720	592	554	554
query18	2048	309	305	305
query19	206	165	165	165
query20	118	117	109	109
query21	211	107	104	104
query22	4700	4410	4629	4410
query23	34361	33797	34003	33797
query24	11059	2732	2528	2528
query25	569	388	380	380
query26	690	154	153	153
query27	2106	281	284	281
query28	5783	2030	2027	2027
query29	729	432	417	417
query30	250	156	149	149
query31	958	762	745	745
query32	95	54	56	54
query33	728	282	291	282
query34	876	454	465	454
query35	966	871	835	835
query36	1093	902	925	902
query37	132	81	80	80
query38	4204	4075	4184	4075
query39	1455	1387	1386	1386
query40	199	121	119	119
query41	48	46	47	46
query42	116	97	95	95
query43	502	447	456	447
query44	1187	812	716	716
query45	236	191	194	191
query46	1077	765	736	736
query47	1892	1779	1789	1779
query48	375	295	296	295
query49	820	421	428	421
query50	791	404	409	404
query51	6766	6726	6745	6726
query52	103	93	90	90
query53	246	177	177	177
query54	905	442	447	442
query55	74	76	71	71
query56	264	246	236	236
query57	1160	1065	1074	1065
query58	225	225	248	225
query59	2820	2644	2722	2644
query60	284	252	258	252
query61	96	94	93	93
query62	777	649	656	649
query63	202	182	179	179
query64	9211	2270	1736	1736
query65	3205	3129	3120	3120
query66	736	327	330	327
query67	15519	14961	14910	14910
query68	7953	550	544	544
query69	439	401	418	401
query70	1154	1128	1067	1067
query71	586	277	268	268
query72	19704	16138	16460	16138
query73	820	326	322	322
query74	9017	8762	8836	8762
query75	5120	2664	2712	2664
query76	5058	1045	956	956
query77	757	299	306	299
query78	9769	9012	9866	9012
query79	8349	514	524	514
query80	1053	487	486	486
query81	588	222	232	222
query82	786	131	136	131
query83	324	190	143	143
query84	269	76	75	75
query85	1300	281	275	275
query86	416	295	294	294
query87	4659	4600	4565	4565
query88	4935	2495	2489	2489
query89	440	290	280	280
query90	1994	189	190	189
query91	119	95	95	95
query92	64	50	47	47
query93	5634	527	528	527
query94	884	301	295	295
query95	378	263	264	263
query96	618	277	278	277
query97	3227	3055	3057	3055
query98	220	206	197	197
query99	1510	1263	1287	1263
Total cold run time: 326655 ms
Total hot run time: 201558 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

ClickBench: Total hot run time: 30.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d00c72196ea02ddb2c9bd2b4839be052d553b98f, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.69	0.07	0.07
query5	0.52	0.50	0.47
query6	1.13	0.72	0.74
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.48
query10	0.56	0.54	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.59	0.61	0.59
query14	0.76	0.77	0.79
query15	0.86	0.82	0.82
query16	0.36	0.40	0.37
query17	0.98	1.00	1.03
query18	0.23	0.21	0.21
query19	1.81	1.74	1.77
query20	0.01	0.01	0.01
query21	15.40	0.75	0.65
query22	4.39	7.82	1.47
query23	18.33	1.34	1.34
query24	2.12	0.22	0.22
query25	0.16	0.08	0.08
query26	0.29	0.22	0.21
query27	0.45	0.23	0.22
query28	13.23	1.03	0.99
query29	12.59	3.29	3.26
query30	0.24	0.05	0.05
query31	2.89	0.38	0.38
query32	3.28	0.48	0.47
query33	2.88	2.88	2.92
query34	17.21	4.36	4.38
query35	4.53	4.41	4.40
query36	0.65	0.46	0.46
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.12	0.13
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.17 s
Total hot run time: 30.21 s

@w41ter
[chore](table) Add batch method to get visible version of the olap table
18c6290 
Since get visible version is a heavy operation in the cloud mode, this
PR add a batch method, to obtain all visible versions via only one RPC.
@w41ter w41ter force-pushed the batch_get_table_versions branch from d00c721 to 18c6290 Compare August 12, 2024 07:27
@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 12, 2024

run buildall

@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Aug 12, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Aug 12, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 12, 2024

TPC-H: Total hot run time: 39425 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18c62901219b02135a3cb93230340942dad677d9, data reload: false

------ Round 1 ----------------------------------
q1	17983	4508	4392	4392
q2	2775	182	197	182
q3	11101	1190	1087	1087
q4	10441	769	761	761
q5	7694	2542	2482	2482
q6	226	137	138	137
q7	979	597	601	597
q8	9492	1908	1899	1899
q9	9630	6529	6543	6529
q10	7048	2234	2156	2156
q11	450	249	243	243
q12	461	217	216	216
q13	17956	2985	3001	2985
q14	280	232	240	232
q15	525	482	475	475
q16	507	384	376	376
q17	973	704	662	662
q18	7961	7489	7381	7381
q19	3598	1111	980	980
q20	690	319	346	319
q21	5261	4337	4414	4337
q22	1111	1003	997	997
Total cold run time: 117142 ms
Total hot run time: 39425 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4467	4243	4205	4205
q2	382	283	269	269
q3	2864	2612	2566	2566
q4	1862	1633	1567	1567
q5	5227	5240	5257	5240
q6	221	129	130	129
q7	2045	1670	1653	1653
q8	3159	3340	3297	3297
q9	8396	8353	8406	8353
q10	3390	3159	3125	3125
q11	594	487	481	481
q12	771	571	603	571
q13	17343	2979	2938	2938
q14	296	277	284	277
q15	518	478	483	478
q16	463	420	413	413
q17	1765	1481	1462	1462
q18	7596	7440	7452	7440
q19	1645	1580	1659	1580
q20	1965	1797	1780	1780
q21	5321	5191	5107	5107
q22	1056	991	1032	991
Total cold run time: 71346 ms
Total hot run time: 53922 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 12, 2024

TPC-DS: Total hot run time: 200102 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18c62901219b02135a3cb93230340942dad677d9, data reload: false

query1	917	370	358	358
query2	6454	2022	1865	1865
query3	6652	206	211	206
query4	33696	23215	23008	23008
query5	4201	489	510	489
query6	272	167	165	165
query7	4572	288	287	287
query8	235	197	192	192
query9	8565	2472	2479	2472
query10	520	460	444	444
query11	16085	15067	14919	14919
query12	151	101	96	96
query13	1619	389	357	357
query14	10101	7580	7624	7580
query15	262	226	211	211
query16	7725	498	479	479
query17	1739	575	552	552
query18	1920	298	285	285
query19	192	151	146	146
query20	114	107	106	106
query21	208	105	102	102
query22	4411	4122	3990	3990
query23	33891	33077	32946	32946
query24	12029	2696	2590	2590
query25	638	381	374	374
query26	1784	150	148	148
query27	3056	274	279	274
query28	7712	2068	2062	2062
query29	1051	392	401	392
query30	300	147	143	143
query31	984	698	718	698
query32	93	52	56	52
query33	739	286	280	280
query34	982	457	466	457
query35	943	793	798	793
query36	1102	927	938	927
query37	300	83	78	78
query38	4252	4067	4158	4067
query39	1416	1358	1410	1358
query40	284	116	114	114
query41	47	44	44	44
query42	109	92	100	92
query43	506	465	452	452
query44	1262	720	729	720
query45	238	207	202	202
query46	1081	708	714	708
query47	1856	1753	1778	1753
query48	369	293	301	293
query49	1193	418	412	412
query50	795	408	415	408
query51	6818	6724	6642	6642
query52	99	90	90	90
query53	249	180	179	179
query54	970	443	465	443
query55	73	72	71	71
query56	273	246	241	241
query57	1134	1063	1035	1035
query58	246	223	237	223
query59	3015	2793	2890	2793
query60	285	260	258	258
query61	99	90	99	90
query62	830	647	664	647
query63	208	181	178	178
query64	10536	2279	1745	1745
query65	3222	3170	3172	3170
query66	1365	349	334	334
query67	15196	14816	14749	14749
query68	4535	551	534	534
query69	448	365	403	365
query70	1113	1072	1058	1058
query71	397	270	272	270
query72	18054	16569	16150	16150
query73	750	324	327	324
query74	9191	8858	8757	8757
query75	3452	2682	2693	2682
query76	2798	969	1032	969
query77	453	308	363	308
query78	10193	9490	9109	9109
query79	2083	513	523	513
query80	1145	500	490	490
query81	587	234	219	219
query82	718	135	132	132
query83	207	143	192	143
query84	223	78	77	77
query85	1987	264	266	264
query86	521	288	283	283
query87	4714	4454	4546	4454
query88	4425	2489	2506	2489
query89	396	288	294	288
query90	1843	192	192	192
query91	120	92	95	92
query92	122	47	50	47
query93	1949	538	533	533
query94	948	295	297	295
query95	348	261	261	261
query96	595	280	270	270
query97	3187	3038	3013	3013
query98	209	199	203	199
query99	1555	1296	1288	1288
Total cold run time: 313507 ms
Total hot run time: 200102 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 12, 2024

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18c62901219b02135a3cb93230340942dad677d9, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.06	0.07
query5	0.49	0.49	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.52
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.11
query13	0.59	0.60	0.58
query14	0.76	0.77	0.77
query15	0.85	0.82	0.81
query16	0.36	0.37	0.37
query17	1.03	1.01	0.95
query18	0.23	0.22	0.22
query19	1.80	1.75	1.83
query20	0.02	0.02	0.01
query21	15.39	0.74	0.64
query22	4.08	8.22	1.63
query23	18.30	1.35	1.36
query24	2.14	0.23	0.21
query25	0.16	0.08	0.09
query26	0.30	0.21	0.22
query27	0.46	0.22	0.22
query28	13.26	1.01	0.98
query29	12.62	3.31	3.27
query30	0.24	0.05	0.05
query31	2.88	0.39	0.38
query32	3.27	0.48	0.46
query33	2.87	2.90	2.89
query34	17.11	4.30	4.46
query35	4.45	4.41	4.38
query36	0.66	0.49	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.66 s
Total hot run time: 30.26 s

dataroaring
dataroaring approved these changes Aug 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Aug 12, 2024

PR approved by at least one committer and no changes requested.

@w41ter w41ter merged commit 285f68a into apache:master Aug 12, 2024
@w41ter w41ter deleted the batch_get_table_versions branch August 12, 2024 10:01
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@w41ter @wyxxxcat
[chore](table) Add batch method to get visible version of the olap ta…
41c2973 
…ble (apache#38949)

Since get visible version is a heavy operation in the cloud mode, this
PR add a batch method, to obtain all visible versions via only one RPC.
@zddr zddr mentioned this pull request Aug 16, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 16, 2024
@w41ter
[chore](table) Add batch method to get visible version of the olap ta…
f40c664 
…ble (apache#38949)
@w41ter w41ter mentioned this pull request Aug 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 17, 2024
@w41ter
[chore](table) Add batch method to get visible version of the olap ta…
fd4d1f4 
…ble (#38949) (#39495)

Cherry-pick #38949
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@w41ter @dataroaring
[chore](table) Add batch method to get visible version of the olap ta…
d115a9f 
…ble (#38949)

Since get visible version is a heavy operation in the cloud mode, this
PR add a batch method, to obtain all visible versions via only one RPC.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zddr zddr zddr approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@w41ter @doris-robot @zddr @dataroaring @yiguolei