[enhance](mtmv)Mv refresh on commit by zddr · Pull Request #34548 · apache/doris

zddr · 2024-05-08T10:57:25Z

Proposed changes

Issue Number: close #xxx

support refresh MTMV when base table data change / drop partition/replace partition

CREATE MATERIALIZED VIEW mv1
        REFRESH ON COMMIT
        AS 
        SELECT xxx;

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-08T10:57:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

…esh_by_base

doris-robot · 2024-05-27T08:18:55Z

ClickBench: Total hot run time: 29.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93a834d3386d065ebfa8ad56185bd527bbf83ea7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.50	0.50	0.49
query6	1.11	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.53	0.48	0.48
query10	0.54	0.56	0.53
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.78	0.78
query15	0.82	0.81	0.80
query16	0.37	0.37	0.37
query17	0.98	1.02	1.02
query18	0.22	0.23	0.25
query19	1.86	1.73	1.82
query20	0.01	0.01	0.01
query21	15.45	0.70	0.68
query22	4.65	7.89	1.02
query23	18.29	1.30	1.27
query24	1.62	0.33	0.24
query25	0.14	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.27	1.00	0.98
query29	12.66	3.34	3.27
query30	0.24	0.05	0.05
query31	2.88	0.38	0.38
query32	3.30	0.47	0.46
query33	2.87	2.89	2.87
query34	17.16	4.40	4.47
query35	4.52	4.45	4.54
query36	0.74	0.49	0.49
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.15	0.13
query41	0.08	0.04	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.55 s
Total hot run time: 29.68 s

mymeiyi

LGTM

zddr · 2024-05-27T14:42:10Z

run p0

zddr · 2024-05-28T11:42:22Z

run buildall

zddr · 2024-05-28T12:18:04Z

run buildall

doris-robot · 2024-05-28T13:12:21Z

TPC-H: Total hot run time: 40286 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

------ Round 1 ----------------------------------
q1	18220	4449	4398	4398
q2	2569	205	190	190
q3	11414	1231	1270	1231
q4	10485	822	775	775
q5	7590	2747	2841	2747
q6	217	134	133	133
q7	965	609	606	606
q8	9384	2108	2081	2081
q9	8893	6548	6487	6487
q10	8925	3725	3699	3699
q11	458	248	243	243
q12	442	220	213	213
q13	17760	3017	3017	3017
q14	277	216	232	216
q15	535	477	478	477
q16	512	398	382	382
q17	969	660	718	660
q18	8046	7463	7322	7322
q19	3194	1553	1523	1523
q20	641	322	317	317
q21	5025	3288	3899	3288
q22	345	284	281	281
Total cold run time: 116866 ms
Total hot run time: 40286 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4191	4188	4188
q2	377	266	271	266
q3	3013	2773	2729	2729
q4	1866	1593	1580	1580
q5	5234	5321	5282	5282
q6	217	124	127	124
q7	2131	1778	1791	1778
q8	3209	3350	3286	3286
q9	8354	8439	8364	8364
q10	3864	3720	3663	3663
q11	585	499	498	498
q12	739	606	590	590
q13	17431	3015	2985	2985
q14	286	272	259	259
q15	507	474	477	474
q16	492	425	437	425
q17	1762	1498	1466	1466
q18	7671	7563	7277	7277
q19	2343	1644	1534	1534
q20	2001	1791	1802	1791
q21	4917	4645	4600	4600
q22	602	530	514	514
Total cold run time: 71925 ms
Total hot run time: 53673 ms

doris-robot · 2024-05-28T13:23:19Z

TPC-DS: Total hot run time: 169541 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

query1	939	393	381	381
query2	6467	2616	2342	2342
query3	6650	200	206	200
query4	20114	17436	17267	17267
query5	4122	414	417	414
query6	240	159	151	151
query7	4595	306	291	291
query8	244	183	177	177
query9	8608	2457	2445	2445
query10	453	280	265	265
query11	10588	10062	10326	10062
query12	135	90	88	88
query13	1641	384	371	371
query14	10397	6825	7644	6825
query15	250	169	167	167
query16	7944	269	264	264
query17	1817	523	517	517
query18	2021	284	270	270
query19	198	175	151	151
query20	93	88	85	85
query21	196	133	132	132
query22	4282	3980	4126	3980
query23	33714	33068	33061	33061
query24	11916	2793	2834	2793
query25	681	376	357	357
query26	1808	157	155	155
query27	2904	323	318	318
query28	7273	2094	2096	2094
query29	1093	623	599	599
query30	289	151	148	148
query31	956	783	779	779
query32	100	52	55	52
query33	760	261	266	261
query34	1002	483	470	470
query35	764	623	616	616
query36	1083	956	915	915
query37	174	66	65	65
query38	2866	2772	2797	2772
query39	850	798	804	798
query40	274	133	125	125
query41	46	45	43	43
query42	106	94	97	94
query43	597	588	571	571
query44	1275	737	746	737
query45	186	163	162	162
query46	1075	715	703	703
query47	1858	1763	1790	1763
query48	371	309	302	302
query49	1176	374	379	374
query50	766	405	398	398
query51	6986	6930	6628	6628
query52	104	91	90	90
query53	358	285	299	285
query54	1052	445	432	432
query55	75	72	73	72
query56	274	282	261	261
query57	1152	1026	1039	1026
query58	234	209	210	209
query59	3514	3311	3206	3206
query60	272	263	253	253
query61	92	93	91	91
query62	675	457	472	457
query63	309	285	281	281
query64	9752	2259	1746	1746
query65	3191	3103	3131	3103
query66	1394	342	325	325
query67	15051	15161	14588	14588
query68	4614	560	547	547
query69	437	271	263	263
query70	1187	1114	1137	1114
query71	389	272	271	271
query72	7613	5363	2713	2713
query73	714	323	327	323
query74	6103	5610	5550	5550
query75	3392	2646	2641	2641
query76	2629	987	1000	987
query77	414	266	264	264
query78	10337	9833	9738	9738
query79	1965	522	526	522
query80	1332	435	446	435
query81	516	223	219	219
query82	643	92	91	91
query83	283	168	169	168
query84	234	86	86	86
query85	1772	277	266	266
query86	491	305	305	305
query87	3325	3139	3081	3081
query88	4214	2438	2440	2438
query89	486	397	384	384
query90	1960	199	188	188
query91	132	106	106	106
query92	62	52	50	50
query93	2328	523	510	510
query94	1244	201	198	198
query95	412	317	323	317
query96	598	273	268	268
query97	3181	3069	3011	3011
query98	241	232	220	220
query99	1230	856	844	844
Total cold run time: 276920 ms
Total hot run time: 169541 ms

doris-robot · 2024-05-28T13:28:49Z

ClickBench: Total hot run time: 29.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.07
query5	0.51	0.47	0.50
query6	1.13	0.75	0.72
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.50
query10	0.55	0.56	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.59
query14	0.77	0.78	0.81
query15	0.83	0.81	0.80
query16	0.36	0.37	0.38
query17	1.03	1.02	1.04
query18	0.22	0.22	0.27
query19	1.90	1.73	1.69
query20	0.01	0.01	0.01
query21	15.52	0.70	0.69
query22	4.77	7.82	1.27
query23	18.28	1.38	1.24
query24	1.32	0.46	0.20
query25	0.16	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.31	1.01	0.99
query29	12.69	3.26	3.31
query30	0.24	0.06	0.06
query31	2.88	0.37	0.38
query32	3.28	0.47	0.47
query33	2.91	2.93	2.85
query34	17.07	4.39	4.42
query35	4.57	4.48	4.48
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.55 s
Total hot run time: 29.81 s

zddr · 2024-05-28T14:03:30Z

run feut

zddr · 2024-05-29T02:03:01Z

run feut

github-actions · 2024-05-29T07:44:13Z

PR approved by at least one committer and no changes requested.

support refresh MTMV when base table data change / drop partition/replace partition CREATE MATERIALIZED VIEW mv1 REFRESH ON COMMIT AS SELECT xxx;

pick from master #34548 The modification involving CloudGlobalTransactionMgr was not picked up to 2.1 because the 2.1 branch does not yet have the Thunderbolt CloudGlobalTransactionMgr

support refresh MTMV when base table data change / drop partition/replace partition CREATE MATERIALIZED VIEW mv1 REFRESH ON COMMIT AS SELECT xxx;

zddr added 3 commits May 8, 2024 18:12

1

85a37bd

1

27f241f

1

4cabd48

zddr marked this pull request as draft May 8, 2024 10:57

zddr added 7 commits May 9, 2024 10:33

1

ce5cfb0

1

9a09734

1

3166cc3

1

1ac81df

1

c5b605e

1

9b76f18

1

38df93e

zddr changed the title ~~[enhance](mtmv)Mv refresh by base~~ [enhance](mtmv)Mv refresh on commit May 9, 2024

zddr added 12 commits May 10, 2024 10:35

Merge branch 'master' into mv_refresh_by_base

f96ec9f

1

05dd465

1

38e276b

1

afc56e9

1

84a02bf

1

eae46cf

1

e75150e

1

eac7106

1

b147d25

1

2a59af7

1

ec20ddc

1

88d8ce5

zddr marked this pull request as ready for review May 11, 2024 08:40

zddr added 4 commits May 11, 2024 16:46

1

ff0fda3

Merge remote-tracking branch 'origin/mv_refresh_by_base' into mv_refr…

632f6a9

…esh_by_base

1

c2b7a37

1

a906094

mymeiyi approved these changes May 27, 2024

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 27, 2024

morrySnow marked this pull request as draft May 28, 2024 04:14

1

744704e

zddr marked this pull request as ready for review May 28, 2024 11:42

1

143776a

morrySnow approved these changes May 29, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 29, 2024

morrySnow added the kind/need-document label May 29, 2024

morrySnow merged commit c1676f0 into apache:master May 29, 2024

zddr mentioned this pull request May 31, 2024

[enhance](mtmv)Mv refresh on commit(#34548) #35702

Merged

morrySnow added dev/2.1.4-merged and removed dev/2.1.x labels May 31, 2024

dataroaring pushed a commit that referenced this pull request May 31, 2024

[enhance](mtmv) Mv refresh on commit (#34548)

dd2ed39

support refresh MTMV when base table data change / drop partition/replace partition CREATE MATERIALIZED VIEW mv1 REFRESH ON COMMIT AS SELECT xxx;

gavinchou mentioned this pull request Jun 13, 2024

[Opt](streamload) Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead #36237

Merged

morningman mentioned this pull request Jun 22, 2024

Release Notes 2.1.4 #35752

Closed

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

morningman removed the kind/need-document label Feb 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhance](mtmv)Mv refresh on commit#34548

[enhance](mtmv)Mv refresh on commit#34548
morrySnow merged 38 commits intoapache:masterfrom
zddr:mv_refresh_by_base

zddr commented May 8, 2024 •

edited

Loading

Uh oh!

doris-robot commented May 8, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

mymeiyi left a comment

Uh oh!

zddr commented May 27, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

zddr commented May 29, 2024

Uh oh!

github-actions bot commented May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

zddr commented May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 8, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

mymeiyi left a comment

Choose a reason for hiding this comment

Uh oh!

zddr commented May 27, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

doris-robot commented May 28, 2024

Uh oh!

zddr commented May 28, 2024

Uh oh!

zddr commented May 29, 2024

Uh oh!

github-actions bot commented May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zddr commented May 8, 2024 •

edited

Loading