[pipeline](datagen) Improve datagen operator parallelism by Gabriel39 · Pull Request #37195 · apache/doris

Gabriel39 · 2024-07-03T03:19:29Z

Proposed changes

Now we use DataGenOperator (num_instance=1) -> ResultSinkOperator(num_instance=1) for loading/query tasks.

This PR use a local shuffle to improve its parallelism and the plan is
DataGenOperator (num_instance=1) -> LocalExchangeSink (num_instance=1) -> LocalExchangeSource (num_instance=(cores / 2) -> ResultSinkOperator(num_instance=(cores / 2))

doris-robot · 2024-07-03T03:19:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Gabriel39 · 2024-07-03T03:19:45Z

run buildall

github-actions · 2024-07-03T03:25:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T03:57:08Z

TPC-H: Total hot run time: 39996 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 221f158f011b43e9886566d8359951a2d1076435, data reload: false

------ Round 1 ----------------------------------
q1	17718	4381	4297	4297
q2	2032	192	184	184
q3	10460	1236	1133	1133
q4	10194	760	937	760
q5	7545	2659	2654	2654
q6	228	142	139	139
q7	953	594	609	594
q8	9267	2064	2062	2062
q9	9924	6490	6525	6490
q10	9148	3710	3730	3710
q11	468	231	234	231
q12	455	234	232	232
q13	19105	2967	3043	2967
q14	263	220	219	219
q15	525	483	487	483
q16	521	394	386	386
q17	968	655	677	655
q18	8152	7450	7498	7450
q19	7815	1499	1432	1432
q20	652	317	324	317
q21	4970	3301	3271	3271
q22	385	330	331	330
Total cold run time: 121748 ms
Total hot run time: 39996 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4190	4269	4190
q2	363	276	277	276
q3	3139	2942	2893	2893
q4	1964	1667	1698	1667
q5	5528	5492	5407	5407
q6	226	135	135	135
q7	2277	1825	1819	1819
q8	3293	3418	3385	3385
q9	8784	8788	8742	8742
q10	4177	3704	3856	3704
q11	605	506	512	506
q12	873	680	623	623
q13	16293	3151	3158	3151
q14	304	293	268	268
q15	531	486	502	486
q16	486	432	449	432
q17	1802	1503	1521	1503
q18	8222	7949	7711	7711
q19	1748	1591	1512	1512
q20	2048	1886	1826	1826
q21	7927	4900	4991	4900
q22	631	548	567	548
Total cold run time: 75552 ms
Total hot run time: 55684 ms

doris-robot · 2024-07-03T04:08:26Z

TPC-DS: Total hot run time: 171719 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 221f158f011b43e9886566d8359951a2d1076435, data reload: false

query1	919	374	363	363
query2	6463	2386	2221	2221
query3	6638	206	215	206
query4	19435	17331	17277	17277
query5	3768	471	503	471
query6	290	167	164	164
query7	4594	306	295	295
query8	306	285	293	285
query9	8567	2387	2357	2357
query10	567	307	293	293
query11	10453	10115	9962	9962
query12	117	84	86	84
query13	1660	377	387	377
query14	10588	7595	7716	7595
query15	237	182	184	182
query16	8060	285	284	284
query17	2134	572	556	556
query18	2047	289	305	289
query19	211	162	165	162
query20	97	85	88	85
query21	210	138	131	131
query22	4528	4111	4062	4062
query23	33971	33917	33452	33452
query24	11083	2914	2890	2890
query25	616	422	402	402
query26	703	159	160	159
query27	2305	340	343	340
query28	6151	2172	2146	2146
query29	881	674	656	656
query30	274	157	154	154
query31	984	787	789	787
query32	95	56	60	56
query33	778	324	318	318
query34	898	507	493	493
query35	814	660	676	660
query36	1155	957	940	940
query37	137	78	84	78
query38	2860	2751	2769	2751
query39	848	790	820	790
query40	211	128	126	126
query41	53	52	56	52
query42	122	105	105	105
query43	590	560	553	553
query44	1223	748	758	748
query45	192	168	161	161
query46	1084	726	716	716
query47	1856	1768	1772	1768
query48	386	302	312	302
query49	862	500	416	416
query50	772	386	397	386
query51	6886	6854	6670	6670
query52	100	100	92	92
query53	365	303	291	291
query54	874	468	447	447
query55	74	82	74	74
query56	285	259	269	259
query57	1150	1059	1040	1040
query58	237	240	245	240
query59	3357	3124	3171	3124
query60	306	299	277	277
query61	95	91	93	91
query62	590	440	462	440
query63	328	289	289	289
query64	8825	2272	1740	1740
query65	3142	3096	3112	3096
query66	782	325	333	325
query67	15584	15003	14912	14912
query68	8455	556	552	552
query69	714	437	334	334
query70	1332	1145	1049	1049
query71	513	279	278	278
query72	8884	4951	2714	2714
query73	2022	329	327	327
query74	5885	5572	5498	5498
query75	5188	2655	2716	2655
query76	4938	991	925	925
query77	787	312	296	296
query78	10381	9817	9751	9751
query79	8580	526	523	523
query80	1050	477	472	472
query81	548	221	223	221
query82	562	111	110	110
query83	334	190	174	174
query84	284	84	86	84
query85	1106	343	272	272
query86	386	321	319	319
query87	3281	3092	3094	3092
query88	4060	2464	2461	2461
query89	546	397	413	397
query90	2003	198	192	192
query91	137	102	104	102
query92	62	50	50	50
query93	6770	538	520	520
query94	1268	194	193	193
query95	418	328	328	328
query96	624	273	271	271
query97	3164	3014	3051	3014
query98	217	209	204	204
query99	1130	866	828	828
Total cold run time: 292402 ms
Total hot run time: 171719 ms

doris-robot · 2024-07-03T04:13:58Z

ClickBench: Total hot run time: 30.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 221f158f011b43e9886566d8359951a2d1076435, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.51	0.48
query10	0.56	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.77	0.77
query15	0.84	0.80	0.81
query16	0.36	0.37	0.37
query17	1.02	0.95	1.04
query18	0.23	0.25	0.26
query19	1.85	1.73	1.71
query20	0.01	0.01	0.01
query21	15.49	0.78	0.67
query22	4.35	7.40	2.03
query23	18.24	1.28	1.34
query24	2.14	0.23	0.22
query25	0.16	0.09	0.08
query26	0.25	0.18	0.17
query27	0.08	0.08	0.08
query28	13.24	1.03	0.99
query29	12.64	3.28	3.29
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.28	0.48	0.47
query33	2.84	2.95	2.88
query34	17.24	4.41	4.45
query35	4.47	4.47	4.48
query36	0.66	0.46	0.50
query37	0.19	0.17	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.19	0.14	0.15
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 109.82 s
Total hot run time: 30.71 s

Gabriel39 · 2024-07-04T04:32:14Z

run buildall

github-actions · 2024-07-04T04:36:32Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-04T04:37:18Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-04T06:30:53Z

TPC-H: Total hot run time: 39833 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b86b2ad2fa3130d9c9b79d2fea361befa3385d55, data reload: false

------ Round 1 ----------------------------------
q1	17629	4334	4318	4318
q2	2030	197	193	193
q3	10434	1198	1088	1088
q4	10194	893	794	794
q5	7493	2652	2607	2607
q6	220	138	139	138
q7	947	592	605	592
q8	9220	2082	2056	2056
q9	8728	6521	6485	6485
q10	9053	3732	3734	3732
q11	458	244	246	244
q12	403	241	233	233
q13	17776	3025	2968	2968
q14	266	235	231	231
q15	522	500	483	483
q16	520	385	375	375
q17	966	665	713	665
q18	7980	7384	7421	7384
q19	4281	1416	1396	1396
q20	653	329	320	320
q21	4854	3178	3901	3178
q22	398	353	353	353
Total cold run time: 115025 ms
Total hot run time: 39833 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4266	4236	4236
q2	380	269	264	264
q3	2987	2722	2883	2722
q4	1980	1702	1707	1702
q5	5593	5639	5538	5538
q6	225	140	135	135
q7	2192	1827	1910	1827
q8	3276	3443	3389	3389
q9	8645	8653	8753	8653
q10	4100	3959	3806	3806
q11	571	491	504	491
q12	835	647	633	633
q13	17421	3160	3156	3156
q14	308	267	278	267
q15	543	492	486	486
q16	469	438	434	434
q17	1826	1525	1508	1508
q18	8104	7950	7851	7851
q19	1789	1708	1588	1588
q20	2148	1896	1876	1876
q21	7538	4880	4865	4865
q22	614	537	569	537
Total cold run time: 75933 ms
Total hot run time: 55964 ms

doris-robot · 2024-07-04T06:42:12Z

TPC-DS: Total hot run time: 174348 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b86b2ad2fa3130d9c9b79d2fea361befa3385d55, data reload: false

query1	928	385	370	370
query2	6470	2468	2338	2338
query3	6637	203	216	203
query4	21218	17423	17318	17318
query5	3687	478	489	478
query6	258	170	158	158
query7	4600	290	301	290
query8	324	298	297	297
query9	8537	2467	2427	2427
query10	594	321	275	275
query11	10630	10102	10025	10025
query12	117	90	86	86
query13	1660	382	361	361
query14	8743	7791	7666	7666
query15	257	194	200	194
query16	8032	320	318	318
query17	1819	569	551	551
query18	2090	286	282	282
query19	220	168	171	168
query20	94	87	84	84
query21	213	138	132	132
query22	4343	4067	4031	4031
query23	34010	33717	33562	33562
query24	10724	2933	2916	2916
query25	608	397	418	397
query26	726	163	169	163
query27	2244	332	342	332
query28	6016	2221	2154	2154
query29	895	671	696	671
query30	263	167	153	153
query31	974	804	755	755
query32	98	55	56	55
query33	691	326	321	321
query34	884	476	485	476
query35	754	676	680	676
query36	1120	982	998	982
query37	139	80	82	80
query38	2940	2816	2845	2816
query39	897	869	832	832
query40	211	136	148	136
query41	63	56	56	56
query42	125	108	108	108
query43	634	597	551	551
query44	1122	780	745	745
query45	190	169	169	169
query46	1078	737	727	727
query47	1855	1793	1819	1793
query48	365	305	306	305
query49	953	410	438	410
query50	768	384	392	384
query51	6886	6873	6782	6782
query52	112	90	93	90
query53	352	296	287	287
query54	877	456	453	453
query55	73	71	73	71
query56	285	266	277	266
query57	1104	1036	1039	1036
query58	240	238	277	238
query59	3542	3432	3175	3175
query60	309	293	273	273
query61	99	113	93	93
query62	612	463	439	439
query63	321	295	291	291
query64	8464	2285	1775	1775
query65	3158	3131	3142	3131
query66	749	328	333	328
query67	15832	15060	15243	15060
query68	6465	552	552	552
query69	731	445	349	349
query70	1194	1174	1122	1122
query71	512	282	274	274
query72	8648	5019	5238	5019
query73	820	328	322	322
query74	6000	5549	5564	5549
query75	5322	2674	2708	2674
query76	5007	915	932	915
query77	791	328	299	299
query78	9560	9066	8891	8891
query79	8542	553	539	539
query80	2403	490	497	490
query81	556	229	222	222
query82	791	111	108	108
query83	276	173	168	168
query84	269	88	86	86
query85	1357	295	273	273
query86	437	304	328	304
query87	3304	3124	3107	3107
query88	4613	2390	2369	2369
query89	563	385	402	385
query90	1913	197	192	192
query91	130	106	102	102
query92	59	51	52	51
query93	6943	520	526	520
query94	1164	221	215	215
query95	412	328	329	328
query96	618	269	263	263
query97	3155	3024	3026	3024
query98	232	206	206	206
query99	1155	866	865	865
Total cold run time: 289775 ms
Total hot run time: 174348 ms

doris-robot · 2024-07-04T06:47:43Z

ClickBench: Total hot run time: 30.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b86b2ad2fa3130d9c9b79d2fea361befa3385d55, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.10	0.08
query5	0.51	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.51
query10	0.53	0.55	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.58
query14	0.79	0.78	0.78
query15	0.85	0.81	0.82
query16	0.38	0.37	0.37
query17	0.97	0.99	0.94
query18	0.21	0.23	0.24
query19	1.76	1.67	1.66
query20	0.01	0.02	0.01
query21	15.41	0.79	0.65
query22	4.60	6.75	1.95
query23	18.35	1.44	1.27
query24	2.09	0.23	0.22
query25	0.16	0.08	0.08
query26	0.26	0.18	0.18
query27	0.09	0.08	0.08
query28	13.25	1.02	0.99
query29	12.59	3.26	3.26
query30	0.25	0.07	0.06
query31	2.86	0.38	0.38
query32	3.28	0.47	0.46
query33	2.92	2.90	2.94
query34	16.99	4.39	4.43
query35	4.48	4.49	4.47
query36	0.66	0.48	0.49
query37	0.18	0.16	0.17
query38	0.16	0.15	0.16
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.08	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.65 s
Total hot run time: 30.54 s

github-actions · 2024-07-04T07:16:25Z

clang-tidy review says "All clean, LGTM! 👍"

Gabriel39 · 2024-07-04T07:30:29Z

run buildall

github-actions · 2024-07-04T07:36:07Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-04T08:20:41Z

TPC-H: Total hot run time: 40169 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 524adcd61ad12787ed0638f89cc4428d354aa8ce, data reload: false

------ Round 1 ----------------------------------
q1	17832	5138	4365	4365
q2	2031	197	193	193
q3	10471	1167	1166	1166
q4	10196	829	723	723
q5	7496	2673	2666	2666
q6	225	143	140	140
q7	986	627	602	602
q8	9255	2114	2091	2091
q9	9133	6536	6563	6536
q10	8957	3747	3770	3747
q11	463	240	242	240
q12	417	242	229	229
q13	18940	2988	2973	2973
q14	272	224	224	224
q15	529	494	490	490
q16	527	383	370	370
q17	986	682	745	682
q18	8057	7445	7423	7423
q19	7289	1441	1514	1441
q20	690	315	327	315
q21	4936	3255	3215	3215
q22	401	338	339	338
Total cold run time: 120089 ms
Total hot run time: 40169 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4378	4297	4303	4297
q2	361	279	269	269
q3	2982	2767	2776	2767
q4	1847	1653	1555	1555
q5	5257	5287	5303	5287
q6	219	129	131	129
q7	2124	1809	1772	1772
q8	3218	3365	3367	3365
q9	8384	8317	8347	8317
q10	3893	3676	3679	3676
q11	587	496	512	496
q12	842	599	615	599
q13	17527	2973	3021	2973
q14	292	259	256	256
q15	519	490	468	468
q16	476	414	432	414
q17	1768	1475	1478	1475
q18	7773	7407	7384	7384
q19	1712	1586	1587	1586
q20	1982	1792	1774	1774
q21	4895	4861	4757	4757
q22	631	547	558	547
Total cold run time: 71667 ms
Total hot run time: 54163 ms

doris-robot · 2024-07-04T08:31:45Z

TPC-DS: Total hot run time: 172881 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 524adcd61ad12787ed0638f89cc4428d354aa8ce, data reload: false

query1	917	390	368	368
query2	7133	2433	2346	2346
query3	6658	209	210	209
query4	19184	17311	17354	17311
query5	4178	502	488	488
query6	276	183	186	183
query7	4610	302	293	293
query8	337	299	309	299
query9	8644	2423	2413	2413
query10	619	314	290	290
query11	10504	9995	10037	9995
query12	137	86	88	86
query13	1646	380	374	374
query14	10085	7615	7696	7615
query15	292	187	190	187
query16	8088	315	306	306
query17	1812	551	529	529
query18	2067	279	266	266
query19	223	149	150	149
query20	95	82	84	82
query21	207	140	126	126
query22	4338	4035	3977	3977
query23	33928	33168	33019	33019
query24	11188	2828	2871	2828
query25	604	366	382	366
query26	1572	159	161	159
query27	2955	313	321	313
query28	7205	2083	2081	2081
query29	946	623	625	623
query30	291	154	156	154
query31	963	759	755	755
query32	99	54	56	54
query33	797	289	287	287
query34	946	469	490	469
query35	753	659	637	637
query36	1083	911	936	911
query37	155	78	77	77
query38	2906	2718	2757	2718
query39	901	823	798	798
query40	276	128	126	126
query41	54	53	57	53
query42	122	104	98	98
query43	593	552	547	547
query44	1242	723	724	723
query45	200	169	163	163
query46	1073	717	732	717
query47	1875	1754	1778	1754
query48	405	293	310	293
query49	1201	413	419	413
query50	774	382	394	382
query51	6931	6706	6733	6706
query52	106	93	94	93
query53	353	291	297	291
query54	912	443	445	443
query55	75	74	75	74
query56	291	265	272	265
query57	1115	1081	1047	1047
query58	259	266	256	256
query59	3415	3087	3266	3087
query60	312	295	284	284
query61	104	95	95	95
query62	701	444	446	444
query63	327	302	293	293
query64	9898	2274	1764	1764
query65	3131	3076	3129	3076
query66	1352	343	337	337
query67	15516	14903	14883	14883
query68	4823	552	559	552
query69	640	445	346	346
query70	1220	1197	1124	1124
query71	468	287	280	280
query72	7022	5555	5415	5415
query73	765	331	325	325
query74	5962	5530	5640	5530
query75	3589	2703	2672	2672
query76	3410	983	950	950
query77	667	312	308	308
query78	9554	8864	8906	8864
query79	2684	525	529	525
query80	2054	497	490	490
query81	552	232	231	231
query82	1413	106	107	106
query83	308	178	173	173
query84	275	95	83	83
query85	1491	301	285	285
query86	489	339	286	286
query87	3288	3086	3121	3086
query88	4064	2361	2365	2361
query89	480	403	396	396
query90	1844	194	203	194
query91	146	191	101	101
query92	59	48	50	48
query93	2512	508	508	508
query94	1249	215	219	215
query95	410	311	321	311
query96	596	276	266	266
query97	3211	2968	3040	2968
query98	220	207	200	200
query99	1283	844	834	834
Total cold run time: 278703 ms
Total hot run time: 172881 ms

doris-robot · 2024-07-04T08:37:15Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 524adcd61ad12787ed0638f89cc4428d354aa8ce, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.49	0.49
query6	1.13	0.73	0.74
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.55	0.49	0.48
query10	0.55	0.55	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.77	0.79	0.78
query15	0.85	0.81	0.83
query16	0.35	0.37	0.35
query17	1.00	0.98	1.00
query18	0.23	0.22	0.25
query19	1.77	1.70	1.72
query20	0.01	0.01	0.01
query21	15.41	0.75	0.66
query22	4.49	6.79	1.78
query23	18.32	1.41	1.19
query24	2.16	0.23	0.22
query25	0.16	0.09	0.09
query26	0.26	0.18	0.19
query27	0.08	0.08	0.07
query28	13.19	1.02	1.01
query29	12.60	3.34	3.36
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.26	0.47	0.48
query33	2.87	2.88	2.96
query34	17.13	4.40	4.39
query35	4.49	4.46	4.46
query36	0.66	0.46	0.47
query37	0.19	0.15	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.16	0.15
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.4 s

Gabriel39 · 2024-07-04T10:55:40Z

run buildall

github-actions · 2024-07-04T11:01:29Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-04T11:07:53Z

PR approved by anyone and no changes requested.

github-actions · 2024-07-04T11:11:29Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-07-04T11:15:13Z

TPC-H: Total hot run time: 41567 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 573ce0870fdc8eefe30b19ddfc0338c579d3d36e, data reload: false

------ Round 1 ----------------------------------
q1	17947	4949	4818	4818
q2	2018	192	195	192
q3	10500	1235	1181	1181
q4	10185	814	844	814
q5	7529	2848	2794	2794
q6	239	136	134	134
q7	1029	592	598	592
q8	9251	2242	2277	2242
q9	8840	6880	6849	6849
q10	9001	3831	3863	3831
q11	468	238	236	236
q12	441	238	226	226
q13	17964	2969	2981	2969
q14	285	224	245	224
q15	541	484	485	484
q16	531	401	381	381
q17	1009	593	700	593
q18	8095	7545	7448	7448
q19	6182	1588	1528	1528
q20	665	337	341	337
q21	5062	3358	4136	3358
q22	409	336	362	336
Total cold run time: 118191 ms
Total hot run time: 41567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4848	4782	4640	4640
q2	387	270	271	270
q3	3111	2834	2819	2819
q4	1983	1624	1590	1590
q5	5380	5423	5421	5421
q6	237	134	129	129
q7	2211	1739	1748	1739
q8	3393	3616	3625	3616
q9	8450	8415	8438	8415
q10	3997	3715	3735	3715
q11	599	481	490	481
q12	814	585	613	585
q13	17497	2994	2983	2983
q14	299	268	260	260
q15	530	487	493	487
q16	502	425	418	418
q17	1943	1579	1576	1576
q18	7763	7610	7374	7374
q19	1849	1660	1766	1660
q20	2045	1828	1760	1760
q21	8567	4915	5018	4915
q22	624	536	524	524
Total cold run time: 77029 ms
Total hot run time: 55377 ms

doris-robot · 2024-07-04T11:26:10Z

TPC-DS: Total hot run time: 167780 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 573ce0870fdc8eefe30b19ddfc0338c579d3d36e, data reload: false

query1	930	378	380	378
query2	6464	2443	2328	2328
query3	6648	207	209	207
query4	19122	17416	17317	17317
query5	4173	486	469	469
query6	312	184	172	172
query7	4611	297	288	288
query8	315	288	289	288
query9	8509	2403	2388	2388
query10	595	287	270	270
query11	10633	10053	9973	9973
query12	140	83	83	83
query13	1634	369	352	352
query14	9946	7570	6101	6101
query15	249	192	195	192
query16	7915	309	301	301
query17	1823	557	525	525
query18	1951	271	270	270
query19	194	157	146	146
query20	86	80	79	79
query21	212	122	122	122
query22	4396	4036	4000	4000
query23	33625	33035	33104	33035
query24	12032	2875	2804	2804
query25	664	364	368	364
query26	1807	147	154	147
query27	3072	309	316	309
query28	7852	2075	2063	2063
query29	1137	648	622	622
query30	286	150	156	150
query31	993	719	766	719
query32	96	80	51	51
query33	768	297	287	287
query34	1007	457	462	457
query35	727	652	604	604
query36	1103	919	923	919
query37	284	79	73	73
query38	2839	2748	2739	2739
query39	865	824	804	804
query40	276	121	121	121
query41	55	52	51	51
query42	113	94	106	94
query43	617	567	525	525
query44	1248	716	717	716
query45	193	159	160	159
query46	1070	740	743	740
query47	1851	1775	1797	1775
query48	365	300	300	300
query49	1191	399	411	399
query50	774	379	381	379
query51	6768	6796	6696	6696
query52	105	90	124	90
query53	359	289	287	287
query54	993	445	436	436
query55	72	75	77	75
query56	296	263	300	263
query57	1183	1068	1071	1068
query58	260	243	253	243
query59	3422	3207	2980	2980
query60	337	289	270	270
query61	94	93	94	93
query62	653	434	442	434
query63	316	280	279	279
query64	9862	2259	1726	1726
query65	3146	3076	3102	3076
query66	1351	334	328	328
query67	15543	15312	14805	14805
query68	4737	530	540	530
query69	605	385	323	323
query70	1096	1047	1118	1047
query71	427	282	288	282
query72	7444	4233	2615	2615
query73	759	324	322	322
query74	5894	5496	5557	5496
query75	3574	2670	2651	2651
query76	3200	913	1014	913
query77	654	296	298	296
query78	10266	9058	8965	8965
query79	2130	507	509	507
query80	1590	484	469	469
query81	571	215	216	215
query82	1340	101	106	101
query83	308	172	164	164
query84	268	91	91	91
query85	1485	298	271	271
query86	471	313	317	313
query87	3295	3076	3072	3072
query88	3615	2352	2343	2343
query89	471	375	386	375
query90	1825	186	183	183
query91	128	99	101	99
query92	57	48	48	48
query93	2082	502	496	496
query94	1202	204	270	204
query95	400	306	312	306
query96	587	262	266	262
query97	3172	3066	3008	3008
query98	223	194	201	194
query99	1207	815	850	815
Total cold run time: 277611 ms
Total hot run time: 167780 ms

doris-robot · 2024-07-04T11:31:41Z

ClickBench: Total hot run time: 30.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 573ce0870fdc8eefe30b19ddfc0338c579d3d36e, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.05	0.05
query4	1.66	0.08	0.07
query5	0.50	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.47	0.50
query10	0.53	0.54	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.78	0.78	0.78
query15	0.83	0.80	0.82
query16	0.35	0.36	0.37
query17	1.01	1.04	1.01
query18	0.22	0.25	0.26
query19	1.84	1.70	1.68
query20	0.01	0.00	0.01
query21	15.42	0.76	0.65
query22	4.58	7.60	1.77
query23	18.28	1.37	1.21
query24	2.17	0.23	0.22
query25	0.16	0.10	0.09
query26	0.30	0.22	0.21
query27	0.45	0.22	0.22
query28	13.20	1.02	1.02
query29	12.60	3.28	3.24
query30	0.26	0.06	0.05
query31	2.87	0.38	0.38
query32	3.27	0.47	0.47
query33	2.86	2.93	2.88
query34	17.07	4.37	4.34
query35	4.43	4.39	4.46
query36	0.65	0.46	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.99 s
Total hot run time: 30.3 s

Now we use ``` DataGenOperator (num_instance=1) -> ResultSinkOperator(num_instance=1) ``` for loading/query tasks. This PR use a local shuffle to improve its parallelism and the plan is ``` DataGenOperator (num_instance=1) -> LocalExchangeSink (num_instance=1) -> LocalExchangeSource (num_instance=(cores / 2) -> ResultSinkOperator(num_instance=(cores / 2)) ```

pick #34122 #35454 #35716 #37195

Gabriel39 added 2 commits July 4, 2024 12:05

[pipeline](datagen) Improve datagen operator parallism

e566eec

[pipeline](datagen) Improve datagen operator parallism

6f4469f

Gabriel39 force-pushed the imp_0703 branch from 221f158 to 6f4469f Compare July 4, 2024 04:30

update

b86b2ad

update

3019451

update

524adcd

Gabriel39 changed the title ~~[pipeline](datagen) Improve datagen operator parallism~~ [pipeline](datagen) Improve datagen operator parallelism Jul 4, 2024

update

573ce08

Mryange approved these changes Jul 4, 2024

View reviewed changes

github-actions bot added the reviewed label Jul 4, 2024

BiteTheDDDDt approved these changes Jul 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024

Gabriel39 merged commit 8989e92 into apache:master Jul 5, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

This was referenced Aug 28, 2024

[cherry-pick](branch-2.1) Improve local shuffle strategy #40030

Merged

[cherry-pick](branch-2.1) Improve local shuffle strategy #40038

Closed

Gabriel39 added a commit that referenced this pull request Aug 29, 2024

[cherry-pick](branch-2.1) Improve local shuffle strategy (#40030)

7155711

pick #34122 #35454 #35716 #37195

Conversation

Gabriel39 commented Jul 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

Gabriel39 commented Jul 3, 2024

Uh oh!

github-actions bot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

Gabriel39 commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

Gabriel39 commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

Gabriel39 commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Gabriel39 commented Jul 3, 2024 •

edited

Loading