[feature](cloud) Support multi follower fe in cloud by deardeng · Pull Request #38388 · apache/doris

deardeng · 2024-07-25T12:50:23Z

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-07-25T12:50:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

deardeng · 2024-07-25T12:50:46Z

run buildall

github-actions · 2024-07-25T12:56:51Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-25T13:54:14Z

TPC-H: Total hot run time: 39963 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

------ Round 1 ----------------------------------
q1	17643	4640	4369	4369
q2	2360	200	195	195
q3	10513	1264	1099	1099
q4	10146	775	786	775
q5	7546	2788	2713	2713
q6	216	138	139	138
q7	1017	622	596	596
q8	9224	1963	1917	1917
q9	9268	6994	6936	6936
q10	8751	3773	3821	3773
q11	491	251	250	250
q12	394	225	222	222
q13	17889	2976	2952	2952
q14	277	244	241	241
q15	536	476	479	476
q16	505	394	382	382
q17	979	663	670	663
q18	8174	7608	7342	7342
q19	4936	1048	1039	1039
q20	670	347	341	341
q21	5047	3251	4025	3251
q22	351	303	293	293
Total cold run time: 116933 ms
Total hot run time: 39963 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4502	4225	4225	4225
q2	371	264	277	264
q3	3011	2789	2734	2734
q4	1880	1613	1609	1609
q5	5298	5340	5334	5334
q6	218	134	134	134
q7	2119	1762	1740	1740
q8	3218	3373	3320	3320
q9	8925	8923	8841	8841
q10	3900	3674	3716	3674
q11	617	490	495	490
q12	785	615	605	605
q13	17453	3006	2968	2968
q14	303	269	268	268
q15	521	470	485	470
q16	469	431	422	422
q17	1778	1495	1468	1468
q18	7645	7503	7497	7497
q19	1663	1493	1514	1493
q20	1996	1793	1781	1781
q21	4899	4579	4787	4579
q22	576	518	542	518
Total cold run time: 72147 ms
Total hot run time: 54434 ms

doris-robot · 2024-07-25T14:05:20Z

TPC-DS: Total hot run time: 172269 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

query1	917	387	367	367
query2	6446	1953	1940	1940
query3	6657	204	217	204
query4	26941	17554	17471	17471
query5	4255	489	479	479
query6	280	170	162	162
query7	4585	288	286	286
query8	241	196	194	194
query9	8455	2431	2406	2406
query10	443	278	264	264
query11	10579	10044	10100	10044
query12	139	84	83	83
query13	1604	366	369	366
query14	10499	7920	7184	7184
query15	224	165	162	162
query16	7171	468	485	468
query17	1328	536	544	536
query18	1828	279	273	273
query19	186	139	140	139
query20	93	84	89	84
query21	206	98	101	98
query22	4236	4087	4015	4015
query23	33738	33191	33257	33191
query24	11247	2841	2944	2841
query25	645	371	377	371
query26	1341	155	152	152
query27	3030	267	275	267
query28	7359	2026	2015	2015
query29	899	413	436	413
query30	284	153	150	150
query31	963	741	740	740
query32	97	52	56	52
query33	764	345	320	320
query34	900	473	475	473
query35	907	730	732	730
query36	1124	939	932	932
query37	151	79	80	79
query38	2825	2752	2734	2734
query39	899	803	787	787
query40	277	116	120	116
query41	46	44	44	44
query42	117	104	97	97
query43	494	472	472	472
query44	1224	724	733	724
query45	209	173	175	173
query46	1094	717	755	717
query47	1833	1738	1752	1738
query48	373	306	295	295
query49	1131	436	446	436
query50	809	412	414	412
query51	6801	6769	6779	6769
query52	140	90	88	88
query53	257	179	182	179
query54	853	441	441	441
query55	76	79	74	74
query56	299	279	279	279
query57	1168	1040	1055	1040
query58	265	257	271	257
query59	2848	2776	2720	2720
query60	324	274	287	274
query61	97	95	92	92
query62	831	646	635	635
query63	209	189	188	188
query64	10158	2243	1711	1711
query65	3168	3128	3116	3116
query66	1037	355	323	323
query67	15381	14801	14748	14748
query68	4576	554	565	554
query69	441	310	306	306
query70	1085	1156	1017	1017
query71	418	273	263	263
query72	7539	5668	5958	5668
query73	784	327	333	327
query74	6027	5666	5557	5557
query75	3407	2735	2711	2711
query76	2904	942	947	942
query77	460	329	312	312
query78	9697	9315	8829	8829
query79	2707	531	530	530
query80	2135	499	499	499
query81	604	221	218	218
query82	743	138	137	137
query83	313	185	184	184
query84	269	79	79	79
query85	1997	322	307	307
query86	486	315	312	312
query87	3336	3074	3091	3074
query88	3851	2410	2397	2397
query89	403	283	284	283
query90	1791	191	197	191
query91	130	97	106	97
query92	70	52	51	51
query93	2004	552	552	552
query94	815	304	285	285
query95	354	274	269	269
query96	596	270	272	270
query97	3249	3037	2987	2987
query98	229	219	188	188
query99	1602	1282	1243	1243
Total cold run time: 281027 ms
Total hot run time: 172269 ms

doris-robot · 2024-07-25T14:10:50Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.04
query4	1.68	0.07	0.08
query5	0.51	0.48	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.05
query9	0.55	0.49	0.48
query10	0.54	0.56	0.55
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.58
query14	0.74	0.78	0.77
query15	0.85	0.81	0.82
query16	0.37	0.36	0.37
query17	0.96	1.04	0.99
query18	0.23	0.22	0.22
query19	1.77	1.64	1.70
query20	0.01	0.01	0.01
query21	15.42	0.76	0.66
query22	4.90	6.41	2.06
query23	18.24	1.44	1.21
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.24	0.23
query28	13.33	1.01	1.00
query29	12.65	3.32	3.29
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.25	0.47	0.47
query33	2.89	2.90	2.91
query34	17.16	4.34	4.40
query35	4.38	4.45	4.46
query36	0.66	0.46	0.48
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.41 s
Total hot run time: 30.65 s

github-actions · 2024-07-26T02:24:05Z

clang-tidy review says "All clean, LGTM! 👍"

deardeng · 2024-07-26T02:28:11Z

run buildall

yujun777

LGTM

github-actions · 2024-08-20T04:28:37Z

PR approved by anyone and no changes requested.

Co-authored-by: yujun <yu.jun.reach@gmail.com>

github-actions · 2024-09-02T11:55:37Z

clang-tidy review says "All clean, LGTM! 👍"

deardeng · 2024-09-03T02:46:26Z

run buildall

github-actions · 2024-09-03T02:51:57Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-09-03T04:19:39Z

TPC-H: Total hot run time: 38092 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

------ Round 1 ----------------------------------
q1	18047	4526	4353	4353
q2	2038	181	185	181
q3	11489	993	1198	993
q4	10177	722	747	722
q5	7747	2878	2788	2788
q6	239	138	142	138
q7	969	626	597	597
q8	9365	2106	2053	2053
q9	7063	6614	6637	6614
q10	6983	2241	2209	2209
q11	462	245	242	242
q12	394	219	221	219
q13	17750	3054	3096	3054
q14	271	230	238	230
q15	542	478	493	478
q16	583	536	515	515
q17	999	626	731	626
q18	7532	6886	6949	6886
q19	1410	1104	999	999
q20	706	337	338	337
q21	3944	2916	2844	2844
q22	1119	1033	1014	1014
Total cold run time: 109829 ms
Total hot run time: 38092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4348	4263	4263
q2	382	278	267	267
q3	2896	2670	2621	2621
q4	1903	1670	1668	1668
q5	5634	5697	5745	5697
q6	244	138	135	135
q7	2256	1809	1799	1799
q8	3286	3445	3448	3445
q9	8910	8870	8828	8828
q10	3571	3360	3429	3360
q11	638	516	529	516
q12	827	673	655	655
q13	14556	3168	3284	3168
q14	318	297	299	297
q15	544	492	475	475
q16	611	587	568	568
q17	1848	1582	1521	1521
q18	8260	8005	7867	7867
q19	1724	1496	1442	1442
q20	2140	1895	1920	1895
q21	5707	5326	5360	5326
q22	1145	1062	1026	1026
Total cold run time: 71802 ms
Total hot run time: 56839 ms

doris-robot · 2024-09-03T04:31:53Z

TPC-DS: Total hot run time: 192673 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

query1	1247	907	860	860
query2	6343	2046	1993	1993
query3	10755	3981	3896	3896
query4	60186	26974	23422	23422
query5	5313	506	500	500
query6	401	157	158	157
query7	5788	286	298	286
query8	294	213	238	213
query9	8809	2441	2455	2441
query10	499	275	262	262
query11	18165	15114	15525	15114
query12	153	103	96	96
query13	1589	402	411	402
query14	11116	7304	7531	7304
query15	221	180	185	180
query16	7388	499	487	487
query17	1105	604	553	553
query18	1746	291	297	291
query19	299	148	143	143
query20	127	105	106	105
query21	209	110	107	107
query22	4554	4511	4816	4511
query23	34060	33646	33188	33188
query24	5954	2869	2867	2867
query25	546	392	380	380
query26	685	152	154	152
query27	1795	276	277	276
query28	3915	2030	2022	2022
query29	627	404	403	403
query30	242	157	159	157
query31	909	766	779	766
query32	83	49	54	49
query33	474	281	303	281
query34	856	483	484	483
query35	804	700	696	696
query36	1072	954	926	926
query37	144	89	88	88
query38	4043	3834	3834	3834
query39	1448	1401	1371	1371
query40	199	115	113	113
query41	45	45	45	45
query42	112	92	98	92
query43	510	462	469	462
query44	1106	722	719	719
query45	196	206	167	167
query46	1078	743	779	743
query47	1869	1796	1801	1796
query48	360	300	298	298
query49	747	415	428	415
query50	806	418	403	403
query51	7036	6837	6780	6780
query52	96	87	85	85
query53	246	174	179	174
query54	559	459	458	458
query55	74	75	73	73
query56	272	252	246	246
query57	1202	1074	1045	1045
query58	227	250	237	237
query59	3116	2914	2828	2828
query60	291	269	268	268
query61	102	98	102	98
query62	746	641	622	622
query63	210	183	189	183
query64	2581	671	661	661
query65	3237	3155	3124	3124
query66	628	340	334	334
query67	15460	15269	15142	15142
query68	2973	560	546	546
query69	399	290	276	276
query70	1168	1153	1029	1029
query71	332	286	275	275
query72	6184	4056	4022	4022
query73	756	329	326	326
query74	9157	8952	9045	8952
query75	3346	2696	2729	2696
query76	1359	999	1007	999
query77	564	323	331	323
query78	9612	9637	9065	9065
query79	1048	525	519	519
query80	711	526	519	519
query81	469	237	237	237
query82	250	151	147	147
query83	179	230	148	148
query84	251	72	74	72
query85	674	297	279	279
query86	322	305	295	295
query87	4332	4296	4351	4296
query88	3173	2316	2317	2316
query89	374	286	288	286
query90	1995	190	188	188
query91	124	101	100	100
query92	59	50	50	50
query93	1029	525	529	525
query94	676	286	291	286
query95	322	297	253	253
query96	584	261	273	261
query97	3232	3061	3065	3061
query98	216	195	199	195
query99	1518	1258	1300	1258
Total cold run time: 304463 ms
Total hot run time: 192673 ms

doris-robot · 2024-09-03T04:37:30Z

ClickBench: Total hot run time: 32.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.53	0.50	0.50
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.48	0.48
query10	0.56	0.54	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.60	0.59
query14	2.04	2.10	2.07
query15	0.86	0.82	0.82
query16	0.37	0.38	0.38
query17	1.05	1.01	1.06
query18	0.21	0.19	0.21
query19	1.93	1.83	1.81
query20	0.01	0.02	0.01
query21	15.41	0.67	0.65
query22	4.00	7.09	2.58
query23	18.26	1.38	1.27
query24	2.08	0.24	0.21
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.26	1.02	1.00
query29	12.64	3.31	3.27
query30	0.24	0.06	0.05
query31	2.86	0.40	0.39
query32	3.26	0.48	0.47
query33	2.99	3.02	3.02
query34	17.02	4.45	4.51
query35	4.53	4.48	4.53
query36	0.66	0.48	0.50
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.05
query40	0.14	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.75 s
Total hot run time: 32.85 s

github-actions · 2024-09-05T02:07:14Z

PR approved by at least one committer and no changes requested.

github-actions bot added the meta-change label Jul 25, 2024

yujun777 reviewed Jul 30, 2024

View reviewed changes

Comment thread regression-test/suites/cloud_p0/multi_cluster/test_multi_follower.groovy Outdated

w41ter reviewed Jul 31, 2024

View reviewed changes

Comment thread gensrc/proto/cloud.proto

Comment thread fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java Outdated

dataroaring added the dev/3.0.x label Aug 13, 2024

yujun777 approved these changes Aug 20, 2024

View reviewed changes

github-actions bot added the reviewed label Aug 20, 2024

gavinchou reviewed Aug 30, 2024

View reviewed changes

Comment thread cloud/src/resource-manager/resource_manager.cpp Outdated

gavinchou requested changes Sep 1, 2024

View reviewed changes

deardeng and others added 3 commits September 2, 2024 17:34

[feature](cloud) Support multi follower fe in cloud

41a09f6

Co-authored-by: yujun <yu.jun.reach@gmail.com>

fix cloud ut

2b3f6af

fix review

2a63d1f

deardeng force-pushed the cloud-add-follower branch from 7329fa9 to 2a63d1f Compare September 2, 2024 11:48

fix review

63cff66

w41ter approved these changes Sep 4, 2024

View reviewed changes

gavinchou approved these changes Sep 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024

dataroaring merged commit 0013c31 into apache:master Sep 5, 2024

gavinchou pushed a commit that referenced this pull request Sep 10, 2024

[feature](cloud) Support multi follower fe in cloud (#38388)

0e11380

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Oct 12, 2024

gavinchou mentioned this pull request Oct 13, 2024

Release Note 3.0.2 #41558

Closed

Conversation

deardeng commented Jul 25, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 25, 2024

Uh oh!

deardeng commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 25, 2024

Uh oh!

doris-robot commented Jul 25, 2024

Uh oh!

doris-robot commented Jul 25, 2024

Uh oh!

doris-robot commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 26, 2024

Uh oh!

deardeng commented Jul 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024

Uh oh!

deardeng commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants