Skip to content

[fix](mtmv)fix mtmv task nereids cost too much time#37589

Merged
morrySnow merged 2 commits intoapache:masterfrom
zddr:fix_mtmv_nereids_cost_too_much
Jul 11, 2024
Merged

[fix](mtmv)fix mtmv task nereids cost too much time#37589
morrySnow merged 2 commits intoapache:masterfrom
zddr:fix_mtmv_nereids_cost_too_much

Conversation

@zddr
Copy link
Copy Markdown
Contributor

@zddr zddr commented Jul 10, 2024

cause: when mtmv task need refresh multi partition, will be split into multiple insert overwrite tasks,they use same ConnectContext and StatementContext,nereids time is calculated based on StatementContext, so multiple tasks will accumulate time

fix: each insert overwrite tasks use unique StatementContext

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Copy Markdown
Contributor Author

zddr commented Jul 10, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 39970 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4767136c1fd1c20cd9ebd40008bced308dd6196b, data reload: false

------ Round 1 ----------------------------------
q1	17621	4390	4287	4287
q2	2026	189	182	182
q3	10457	1245	1047	1047
q4	10201	765	790	765
q5	7546	2633	2642	2633
q6	220	141	137	137
q7	953	603	612	603
q8	9221	2072	2092	2072
q9	9002	6576	6542	6542
q10	9030	3776	3720	3720
q11	460	229	232	229
q12	445	241	247	241
q13	17781	2983	3007	2983
q14	271	226	224	224
q15	525	489	483	483
q16	523	379	378	378
q17	971	756	674	674
q18	7897	7556	7499	7499
q19	7106	1407	1499	1407
q20	679	331	329	329
q21	4989	3187	4053	3187
q22	425	348	353	348
Total cold run time: 118349 ms
Total hot run time: 39970 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4498	4392	4408	4392
q2	374	285	282	282
q3	3211	3087	3083	3083
q4	2079	1876	1868	1868
q5	5728	5656	5462	5462
q6	224	139	143	139
q7	2260	1908	1834	1834
q8	3344	3496	3421	3421
q9	8700	8752	8677	8677
q10	4163	3688	3834	3688
q11	583	494	496	494
q12	805	652	648	648
q13	16116	3138	3239	3138
q14	298	290	273	273
q15	531	495	499	495
q16	494	428	420	420
q17	1807	1525	1499	1499
q18	8219	8049	7869	7869
q19	1834	1657	1647	1647
q20	2686	1868	1870	1868
q21	9156	4719	4820	4719
q22	632	579	552	552
Total cold run time: 77742 ms
Total hot run time: 56468 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 174127 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4767136c1fd1c20cd9ebd40008bced308dd6196b, data reload: false

query1	896	370	361	361
query2	6359	2550	2417	2417
query3	6636	206	212	206
query4	23773	17510	17409	17409
query5	3778	496	488	488
query6	262	174	188	174
query7	4593	314	283	283
query8	303	299	295	295
query9	8489	2387	2365	2365
query10	439	295	266	266
query11	10693	9930	10056	9930
query12	115	86	83	83
query13	1662	374	383	374
query14	9974	7835	7509	7509
query15	236	186	178	178
query16	7597	305	309	305
query17	1812	555	527	527
query18	1233	271	270	270
query19	192	180	157	157
query20	95	88	83	83
query21	203	130	130	130
query22	4403	4154	4003	4003
query23	34068	33903	33561	33561
query24	10775	2977	2823	2823
query25	618	387	388	387
query26	693	147	147	147
query27	2253	278	274	274
query28	5829	2131	2107	2107
query29	896	657	636	636
query30	277	150	153	150
query31	978	738	789	738
query32	99	52	52	52
query33	656	304	305	304
query34	892	492	499	492
query35	674	591	580	580
query36	1132	979	973	973
query37	143	85	86	85
query38	2973	2870	2825	2825
query39	940	817	789	789
query40	202	122	119	119
query41	54	52	52	52
query42	114	100	101	100
query43	590	543	557	543
query44	1144	739	751	739
query45	207	161	165	161
query46	1085	728	710	710
query47	1846	1764	1767	1764
query48	384	307	293	293
query49	841	427	410	410
query50	780	397	406	397
query51	6785	6717	6770	6717
query52	106	95	90	90
query53	356	285	284	284
query54	861	455	444	444
query55	76	78	74	74
query56	294	254	267	254
query57	1128	1037	1061	1037
query58	245	269	258	258
query59	3416	3190	3260	3190
query60	324	283	285	283
query61	94	93	93	93
query62	815	647	643	643
query63	316	283	280	280
query64	9147	2196	1644	1644
query65	3128	3107	3078	3078
query66	718	332	324	324
query67	15450	15437	15055	15055
query68	4533	535	539	535
query69	523	447	335	335
query70	1189	1110	1158	1110
query71	416	290	274	274
query72	7128	5750	5261	5261
query73	745	328	322	322
query74	5929	5573	5531	5531
query75	3339	2679	2668	2668
query76	2177	950	981	950
query77	462	292	300	292
query78	9581	8908	8852	8852
query79	2638	512	502	502
query80	2406	486	484	484
query81	590	221	213	213
query82	1443	145	136	136
query83	278	165	166	165
query84	257	92	84	84
query85	1398	321	305	305
query86	462	310	321	310
query87	3260	3097	3115	3097
query88	3708	2454	2455	2454
query89	483	387	379	379
query90	1730	188	189	188
query91	131	104	99	99
query92	57	49	52	49
query93	2451	514	510	510
query94	1126	212	213	212
query95	404	317	330	317
query96	600	278	269	269
query97	3203	3009	3119	3009
query98	226	208	199	199
query99	1487	1268	1244	1244
Total cold run time: 272505 ms
Total hot run time: 174127 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 31.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4767136c1fd1c20cd9ebd40008bced308dd6196b, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.69	0.07	0.07
query5	0.48	0.47	0.52
query6	1.16	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.54	0.51	0.49
query10	0.55	0.54	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	0.76	0.78	0.77
query15	0.86	0.81	0.81
query16	0.36	0.38	0.37
query17	1.03	0.98	0.99
query18	0.24	0.23	0.22
query19	1.92	1.84	1.83
query20	0.01	0.01	0.01
query21	15.42	0.77	0.67
query22	4.51	6.05	2.76
query23	18.32	1.45	1.33
query24	2.14	0.23	0.21
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.24
query28	13.24	1.02	1.00
query29	12.58	3.36	3.30
query30	0.26	0.06	0.05
query31	2.87	0.39	0.37
query32	3.30	0.49	0.47
query33	2.85	2.98	2.92
query34	16.84	4.35	4.38
query35	4.47	4.43	4.51
query36	0.65	0.50	0.47
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.03 s
Total hot run time: 31.69 s

@morrySnow morrySnow added the p0_b label Jul 10, 2024
morrySnow
morrySnow reviewed Jul 10, 2024
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java Outdated
@zddr zddr requested a review from morrySnow July 10, 2024 12:30
@zddr
Copy link
Copy Markdown
Contributor Author

zddr commented Jul 10, 2024

run buildall

@wm1581066 wm1581066 added the usercase Important user case type label label Jul 10, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 39942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e9102f1db5ca2194de6bfa43cd68e8e64a9683b, data reload: false

------ Round 1 ----------------------------------
q1	18075	4477	4312	4312
q2	2006	193	188	188
q3	10455	1277	1150	1150
q4	10193	871	715	715
q5	7481	2661	2648	2648
q6	220	138	142	138
q7	964	593	597	593
q8	9314	2067	2067	2067
q9	8914	6550	6541	6541
q10	8801	3752	3676	3676
q11	456	241	241	241
q12	395	226	226	226
q13	17748	2985	3000	2985
q14	261	227	235	227
q15	533	477	497	477
q16	473	398	372	372
q17	973	703	661	661
q18	8038	7706	7522	7522
q19	2863	1462	1513	1462
q20	692	318	312	312
q21	5041	3093	3276	3093
q22	383	336	342	336
Total cold run time: 114279 ms
Total hot run time: 39942 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4340	4279	4240	4240
q2	365	278	268	268
q3	2984	2785	2785	2785
q4	1862	1631	1563	1563
q5	5284	5313	5299	5299
q6	218	132	135	132
q7	2145	1748	1733	1733
q8	3253	3355	3326	3326
q9	8370	8370	8394	8370
q10	3868	3622	3701	3622
q11	602	478	484	478
q12	798	594	607	594
q13	17788	2997	3011	2997
q14	296	266	279	266
q15	526	477	480	477
q16	479	411	429	411
q17	1794	1489	1481	1481
q18	7685	7381	7378	7378
q19	1672	1568	1563	1563
q20	1994	1819	1791	1791
q21	5047	4827	4756	4756
q22	610	550	583	550
Total cold run time: 71980 ms
Total hot run time: 54080 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 174737 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e9102f1db5ca2194de6bfa43cd68e8e64a9683b, data reload: false

query1	903	372	362	362
query2	6468	2501	2432	2432
query3	6676	202	215	202
query4	28399	17567	17464	17464
query5	4248	490	477	477
query6	268	160	164	160
query7	4582	289	283	283
query8	313	288	298	288
query9	8545	2382	2363	2363
query10	450	290	277	277
query11	10409	9999	10046	9999
query12	139	85	89	85
query13	1683	387	375	375
query14	10312	7688	7598	7598
query15	273	187	186	186
query16	7953	327	316	316
query17	1791	564	547	547
query18	2048	285	285	285
query19	209	154	159	154
query20	94	85	85	85
query21	211	124	127	124
query22	4216	4106	3972	3972
query23	33723	33395	33212	33212
query24	11992	2838	2818	2818
query25	676	379	388	379
query26	1805	149	149	149
query27	2914	275	276	275
query28	7779	2072	2062	2062
query29	1063	644	634	634
query30	291	147	153	147
query31	967	726	748	726
query32	94	55	63	55
query33	784	331	309	309
query34	901	494	488	488
query35	691	582	582	582
query36	1083	932	956	932
query37	167	82	81	81
query38	2852	2742	2764	2742
query39	859	823	808	808
query40	282	126	122	122
query41	58	54	54	54
query42	127	100	105	100
query43	577	549	531	531
query44	1223	770	740	740
query45	190	162	169	162
query46	1090	712	757	712
query47	1890	1804	1812	1804
query48	375	311	310	310
query49	1202	426	426	426
query50	774	392	399	392
query51	6865	6813	6825	6813
query52	101	93	99	93
query53	356	295	296	295
query54	920	454	444	444
query55	83	73	73	73
query56	302	261	265	261
query57	1133	1060	1055	1055
query58	264	254	243	243
query59	3396	3437	3290	3290
query60	296	278	274	274
query61	93	94	97	94
query62	840	643	651	643
query63	325	296	294	294
query64	10464	2153	7469	2153
query65	3135	3126	3136	3126
query66	1246	331	332	331
query67	15981	15263	14934	14934
query68	6490	550	539	539
query69	749	437	355	355
query70	1192	1154	1063	1063
query71	487	281	279	279
query72	8679	5438	5525	5438
query73	768	328	331	328
query74	5779	5674	5556	5556
query75	4754	2688	2686	2686
query76	4164	1010	921	921
query77	704	304	311	304
query78	9488	9826	8876	8876
query79	8407	519	537	519
query80	939	476	463	463
query81	580	219	222	219
query82	827	134	138	134
query83	337	164	169	164
query84	279	88	88	88
query85	1449	322	306	306
query86	445	322	313	313
query87	3324	3125	3079	3079
query88	4628	2497	2460	2460
query89	549	379	380	379
query90	2003	191	193	191
query91	132	101	103	101
query92	67	49	50	49
query93	7175	516	506	506
query94	1340	210	211	210
query95	419	323	309	309
query96	626	273	274	273
query97	3189	3004	3019	3004
query98	225	248	194	194
query99	1587	1298	1229	1229
Total cold run time: 304200 ms
Total hot run time: 174737 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e9102f1db5ca2194de6bfa43cd68e8e64a9683b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.68	0.07	0.07
query5	0.50	0.48	0.49
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.58
query14	0.76	0.79	0.77
query15	0.86	0.81	0.82
query16	0.36	0.37	0.37
query17	0.99	1.01	0.95
query18	0.23	0.24	0.23
query19	1.84	1.66	1.71
query20	0.01	0.01	0.01
query21	15.40	0.79	0.66
query22	4.11	7.56	2.16
query23	18.26	1.36	1.40
query24	2.12	0.24	0.23
query25	0.15	0.08	0.09
query26	0.29	0.21	0.21
query27	0.45	0.22	0.23
query28	13.21	1.01	1.00
query29	12.65	3.41	3.36
query30	0.25	0.06	0.05
query31	2.88	0.40	0.40
query32	3.25	0.49	0.46
query33	2.86	2.98	2.84
query34	17.00	4.32	4.34
query35	4.40	4.38	4.41
query36	0.65	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.45 s
Total hot run time: 30.88 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit d15005d into apache:master Jul 11, 2024
zddr added a commit to zddr/incubator-doris that referenced this pull request Jul 15, 2024
@zddr
[fix](mtmv) fix mtmv task nereids cost too much time (apache#37589)
84f7e88 
cause: when mtmv task need refresh multi partition, will be split into
multiple `insert overwrite` tasks,they use same `ConnectContext` and
`StatementContext`,nereids time is calculated based on StatementContext,
so multiple tasks will accumulate time

fix: each `insert overwrite` tasks use unique `StatementContext`
@zddr zddr mentioned this pull request Jul 15, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@zddr
[fix](mtmv) fix mtmv task nereids cost too much time (#37589) (#37819)
f1d22a9 
pick: #37589
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@zddr @seawinde
[fix](mtmv) fix mtmv task nereids cost too much time (apache#37589)
c6adccb 
cause: when mtmv task need refresh multi partition, will be split into
multiple `insert overwrite` tasks,they use same `ConnectContext` and
`StatementContext`,nereids time is calculated based on StatementContext,
so multiple tasks will accumulate time

fix: each `insert overwrite` tasks use unique `StatementContext`
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@zddr @dataroaring
[fix](mtmv) fix mtmv task nereids cost too much time (#37589)
cd3fd50 
cause: when mtmv task need refresh multi partition, will be split into
multiple `insert overwrite` tasks,they use same `ConnectContext` and
`StatementContext`,nereids time is calculated based on StatementContext,
so multiple tasks will accumulate time

fix: each `insert overwrite` tasks use unique `StatementContext`
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

@seawinde seawinde seawinde approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged p0_b reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zddr @doris-robot @morrySnow @seawinde @yiguolei @wm1581066 @dataroaring