[enhance](mtmv)Add a new class MTMVSnapshotIdSnapshot to make the code more readable by zddr · Pull Request #47166 · apache/doris

zddr · 2025-01-17T08:59:47Z

What problem does this PR solve?

before, paimon and iceberg put snapshotId to MTMVVersionSnapshot ,now change to MTMVSnapshotIdSnapshot
compatiblePartitions only consider OlapTable, because other TableType not has history data
Delete constructor methods without id in MTMVVersionSnapshot to avoid misuse

not need pick to 2.1 and 3.0

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Add a new class MTMVSnapshotIdSnapshot to make the code more readable

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-17T08:59:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zddr · 2025-01-17T09:00:09Z

run buildall

doris-robot · 2025-01-17T09:50:53Z

TPC-H: Total hot run time: 32223 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef9badc7d37300719d02282351e8e534acb4c0ae, data reload: false

------ Round 1 ----------------------------------
q1	17578	5640	5424	5424
q2	2060	303	170	170
q3	10501	1268	736	736
q4	10225	971	533	533
q5	7588	2449	2173	2173
q6	199	165	136	136
q7	920	772	614	614
q8	9267	1386	1165	1165
q9	5179	4874	4984	4874
q10	6848	2341	1903	1903
q11	464	264	271	264
q12	356	362	224	224
q13	17761	3713	3138	3138
q14	235	245	208	208
q15	511	485	461	461
q16	627	625	580	580
q17	578	874	333	333
q18	7242	6519	6311	6311
q19	1415	963	536	536
q20	322	330	193	193
q21	2832	2160	1935	1935
q22	382	338	312	312
Total cold run time: 103090 ms
Total hot run time: 32223 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5519	5507	5547	5507
q2	235	341	237	237
q3	2290	2620	2309	2309
q4	1447	1811	1431	1431
q5	4286	4795	4656	4656
q6	172	160	129	129
q7	2100	1993	1819	1819
q8	2664	2835	2668	2668
q9	7254	7267	7248	7248
q10	3110	3268	2808	2808
q11	588	523	494	494
q12	726	790	604	604
q13	3433	3823	3279	3279
q14	295	310	265	265
q15	512	484	472	472
q16	673	661	645	645
q17	1247	1754	1283	1283
q18	7737	7206	7490	7206
q19	861	1137	1067	1067
q20	1991	2022	1896	1896
q21	5767	5341	5076	5076
q22	609	660	570	570
Total cold run time: 53516 ms
Total hot run time: 51669 ms

doris-robot · 2025-01-17T10:02:50Z

TPC-DS: Total hot run time: 189200 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef9badc7d37300719d02282351e8e534acb4c0ae, data reload: false

query1	916	405	383	383
query2	6467	2114	2067	2067
query3	6780	223	216	216
query4	33323	23693	22990	22990
query5	4403	645	481	481
query6	296	210	195	195
query7	4598	494	310	310
query8	302	266	227	227
query9	9557	2580	2573	2573
query10	490	334	245	245
query11	17990	15125	15122	15122
query12	149	103	103	103
query13	1643	512	374	374
query14	10680	7330	6899	6899
query15	215	196	185	185
query16	7998	610	497	497
query17	1715	800	614	614
query18	2187	405	312	312
query19	217	187	158	158
query20	117	116	112	112
query21	219	138	102	102
query22	4726	4484	4517	4484
query23	34374	33994	34357	33994
query24	6657	2314	2424	2314
query25	450	448	382	382
query26	1076	248	154	154
query27	2026	472	319	319
query28	5270	2408	2396	2396
query29	543	515	424	424
query30	230	185	153	153
query31	960	867	793	793
query32	76	60	60	60
query33	510	352	290	290
query34	756	864	533	533
query35	815	844	726	726
query36	1006	1040	949	949
query37	132	101	79	79
query38	4100	4186	4026	4026
query39	1448	1415	1398	1398
query40	204	126	109	109
query41	56	56	56	56
query42	126	106	107	106
query43	525	517	490	490
query44	1328	816	808	808
query45	181	176	167	167
query46	862	1042	644	644
query47	1842	1838	1800	1800
query48	390	410	322	322
query49	792	535	388	388
query50	642	649	400	400
query51	6942	6919	6718	6718
query52	102	99	91	91
query53	222	252	189	189
query54	471	472	416	416
query55	93	84	81	81
query56	248	266	235	235
query57	1160	1143	1091	1091
query58	244	232	240	232
query59	2946	3120	2926	2926
query60	279	275	262	262
query61	137	118	163	118
query62	818	716	650	650
query63	220	186	188	186
query64	4306	1040	674	674
query65	3268	3187	3167	3167
query66	815	408	303	303
query67	16046	15727	15587	15587
query68	5718	823	518	518
query69	464	314	324	314
query70	1238	1145	1125	1125
query71	386	281	256	256
query72	5772	3975	3867	3867
query73	635	759	358	358
query74	10102	9012	8801	8801
query75	3178	3166	2705	2705
query76	3168	1183	782	782
query77	498	373	278	278
query78	10013	10019	9349	9349
query79	3641	810	582	582
query80	682	538	441	441
query81	514	278	234	234
query82	650	155	117	117
query83	176	169	152	152
query84	238	101	79	79
query85	777	367	301	301
query86	395	309	314	309
query87	4583	4553	4419	4419
query88	4948	2177	2112	2112
query89	409	332	294	294
query90	1771	191	194	191
query91	134	135	111	111
query92	67	59	52	52
query93	1960	850	526	526
query94	701	415	311	311
query95	342	276	279	276
query96	492	617	284	284
query97	2822	2867	2733	2733
query98	239	211	206	206
query99	1720	1375	1262	1262
Total cold run time: 289115 ms
Total hot run time: 189200 ms

doris-robot · 2025-01-17T10:08:07Z

ClickBench: Total hot run time: 30.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef9badc7d37300719d02282351e8e534acb4c0ae, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.42	0.42	0.42
query6	1.14	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.51
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.63	0.61	0.59
query14	2.73	2.73	2.71
query15	0.91	0.82	0.82
query16	0.37	0.38	0.37
query17	0.99	1.06	1.07
query18	0.23	0.21	0.22
query19	1.99	1.79	2.01
query20	0.01	0.01	0.01
query21	15.37	0.96	0.57
query22	0.75	0.94	0.60
query23	15.25	1.48	0.58
query24	2.78	0.40	1.73
query25	0.21	0.11	0.15
query26	0.25	0.17	0.15
query27	0.06	0.06	0.05
query28	13.77	1.02	0.45
query29	12.61	3.93	3.34
query30	0.26	0.08	0.08
query31	2.80	0.59	0.38
query32	3.24	0.57	0.45
query33	3.01	3.00	3.03
query34	16.58	5.10	4.55
query35	4.57	4.51	4.50
query36	0.64	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.53 s
Total hot run time: 30.18 s

zddr · 2025-01-21T03:10:41Z

run external

zddr · 2025-01-21T08:27:54Z

run buildall

zddr · 2025-01-21T08:43:32Z

run performance

zddr · 2025-01-21T09:38:08Z

run buildall

doris-robot · 2025-01-21T10:09:42Z

TPC-H: Total hot run time: 32475 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d90dc956f62fff607ee7e30c8c6bbc252fddede4, data reload: false

------ Round 1 ----------------------------------
q1	17585	5508	5441	5441
q2	2051	313	174	174
q3	10417	1265	729	729
q4	10217	987	508	508
q5	7543	2408	2186	2186
q6	197	174	136	136
q7	919	769	624	624
q8	9249	1362	1154	1154
q9	5192	4925	4904	4904
q10	6865	2333	1880	1880
q11	475	273	255	255
q12	349	356	214	214
q13	17775	3707	3126	3126
q14	232	239	211	211
q15	517	477	454	454
q16	637	633	586	586
q17	578	862	336	336
q18	7026	6522	6455	6455
q19	1205	945	547	547
q20	323	339	198	198
q21	2940	2259	2052	2052
q22	372	335	305	305
Total cold run time: 102664 ms
Total hot run time: 32475 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5623	5482	5500	5482
q2	238	330	244	244
q3	2307	2639	2303	2303
q4	1468	1822	1344	1344
q5	4350	4743	4660	4660
q6	168	157	128	128
q7	2100	2010	1793	1793
q8	2682	2818	2719	2719
q9	7328	7163	7213	7163
q10	3025	3222	2746	2746
q11	607	515	499	499
q12	688	766	590	590
q13	3532	3880	3334	3334
q14	275	294	278	278
q15	523	469	466	466
q16	655	679	642	642
q17	1252	1726	1269	1269
q18	7740	7557	7369	7369
q19	898	1162	1093	1093
q20	2023	2011	1888	1888
q21	5687	5338	4968	4968
q22	576	549	546	546
Total cold run time: 53745 ms
Total hot run time: 51524 ms

doris-robot · 2025-01-21T10:21:38Z

TPC-DS: Total hot run time: 188522 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d90dc956f62fff607ee7e30c8c6bbc252fddede4, data reload: false

query1	984	385	362	362
query2	6525	2091	2063	2063
query3	6784	215	212	212
query4	33390	23294	23399	23294
query5	4394	675	463	463
query6	289	193	195	193
query7	4617	519	325	325
query8	301	248	235	235
query9	9669	2599	2574	2574
query10	492	338	288	288
query11	18397	15302	15365	15302
query12	161	111	107	107
query13	1670	536	400	400
query14	11573	7201	7685	7201
query15	237	198	183	183
query16	8450	704	437	437
query17	1641	734	537	537
query18	2094	410	285	285
query19	215	183	186	183
query20	115	110	107	107
query21	206	122	101	101
query22	4326	4094	4073	4073
query23	33873	33019	33168	33019
query24	6171	2212	2396	2212
query25	486	443	379	379
query26	1184	302	157	157
query27	1967	487	327	327
query28	5222	2439	2395	2395
query29	689	563	430	430
query30	228	192	156	156
query31	1017	891	789	789
query32	81	61	62	61
query33	516	382	296	296
query34	751	902	500	500
query35	794	813	752	752
query36	1009	1061	933	933
query37	126	101	79	79
query38	4150	4195	4233	4195
query39	1454	1399	1401	1399
query40	234	121	104	104
query41	53	51	54	51
query42	123	107	106	106
query43	537	546	501	501
query44	1406	797	807	797
query45	179	176	164	164
query46	904	1069	649	649
query47	1802	1816	1776	1776
query48	386	412	332	332
query49	804	495	429	429
query50	668	704	387	387
query51	6995	6879	6702	6702
query52	104	104	92	92
query53	228	264	193	193
query54	523	494	406	406
query55	80	80	81	80
query56	264	267	249	249
query57	1156	1125	1085	1085
query58	248	244	252	244
query59	2968	3180	2973	2973
query60	296	280	251	251
query61	118	113	117	113
query62	818	737	687	687
query63	237	194	189	189
query64	3918	1037	650	650
query65	3230	3138	3149	3138
query66	1059	425	342	342
query67	15913	15507	15309	15309
query68	4871	848	523	523
query69	461	293	263	263
query70	1190	1067	1142	1067
query71	390	290	253	253
query72	5833	3891	3879	3879
query73	642	800	364	364
query74	10253	9075	9026	9026
query75	3258	3178	2664	2664
query76	3219	1273	797	797
query77	540	377	288	288
query78	10008	10035	9456	9456
query79	3303	857	584	584
query80	1684	536	452	452
query81	579	282	247	247
query82	351	152	120	120
query83	270	184	154	154
query84	244	98	70	70
query85	821	353	307	307
query86	481	317	302	302
query87	4434	4489	4345	4345
query88	4973	2148	2139	2139
query89	422	341	291	291
query90	1917	193	192	192
query91	138	141	111	111
query92	68	57	55	55
query93	2773	903	525	525
query94	742	425	296	296
query95	341	273	261	261
query96	498	669	284	284
query97	2766	2859	2708	2708
query98	269	214	203	203
query99	1323	1417	1293	1293
Total cold run time: 290561 ms
Total hot run time: 188522 ms

doris-robot · 2025-01-21T10:26:59Z

ClickBench: Total hot run time: 31.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d90dc956f62fff607ee7e30c8c6bbc252fddede4, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.59	0.11	0.10
query5	0.41	0.42	0.40
query6	1.14	0.64	0.64
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.51	0.50
query10	0.56	0.56	0.56
query11	0.15	0.10	0.10
query12	0.13	0.12	0.11
query13	0.60	0.62	0.59
query14	2.76	2.74	2.79
query15	0.90	0.82	0.82
query16	0.38	0.39	0.40
query17	1.06	1.08	1.07
query18	0.25	0.22	0.21
query19	1.97	2.01	1.81
query20	0.01	0.01	0.01
query21	15.36	0.93	0.60
query22	0.76	0.85	0.75
query23	15.13	1.47	0.62
query24	3.01	1.24	1.66
query25	0.18	0.11	0.18
query26	0.34	0.16	0.14
query27	0.05	0.05	0.05
query28	13.98	1.05	0.44
query29	12.66	3.95	3.27
query30	0.25	0.09	0.07
query31	2.81	0.61	0.39
query32	3.22	0.55	0.46
query33	2.98	2.98	3.10
query34	16.42	5.15	4.47
query35	4.58	4.44	4.50
query36	0.64	0.51	0.51
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.75 s
Total hot run time: 31.09 s

morrySnow

need upgrade / downgrade case?

zfr9527

LGTM

github-actions · 2025-01-24T08:12:23Z

PR approved by anyone and no changes requested.

Jibing-Li

Any problems for upgrading and downgrading?

zddr · 2025-02-13T03:05:33Z

pr of iceberg support MTMV not merge to 2.1 or 3.0.not need compatible

github-actions · 2025-02-14T08:23:57Z

PR approved by at least one committer and no changes requested.

…e more readable (apache#47166) ### What problem does this PR solve? - before, paimon and iceberg put snapshotId to MTMVVersionSnapshot ,now change to MTMVSnapshotIdSnapshot - `compatiblePartitions` only consider OlapTable, because other TableType not has history data - Delete constructor methods without id in MTMVVersionSnapshot to avoid misuse

…e more readable (apache#47166) - before, paimon and iceberg put snapshotId to MTMVVersionSnapshot ,now change to MTMVSnapshotIdSnapshot - `compatiblePartitions` only consider OlapTable, because other TableType not has history data - Delete constructor methods without id in MTMVVersionSnapshot to avoid misuse

Cherry-pick from #43959 #44419 #44415 #44567 #44673 #44998 #45273 #44911 #44726 #45652 #45659 #46257 #46641 #47026 #47166 #48172 #49956 #50979 --------- Co-authored-by: James <lijibing@selectdb.com> Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

1

ef9badc

resolve gson

6af9d0b

Merge branch 'master' into mv_compatible_olap

d90dc95

morrySnow reviewed Jan 22, 2025

View reviewed changes

zfr9527 approved these changes Jan 24, 2025

View reviewed changes

github-actions Bot added the reviewed label Jan 24, 2025

Jibing-Li reviewed Feb 13, 2025

View reviewed changes

morrySnow approved these changes Feb 14, 2025

View reviewed changes

morrySnow added the dev/3.1.x label Feb 14, 2025

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2025

morrySnow merged commit 18fe279 into apache:master Feb 14, 2025

morrySnow added dev/3.1.x and removed dev/3.1.x labels Jun 19, 2025

github-actions Bot added the dev/3.1.x-conflict label Jun 19, 2025

zddr mentioned this pull request Jun 20, 2025

branch-3.1:[enhance](mtmv)MTMV support iceberg/hudi/paimon #52040

Merged

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jun 23, 2025

Conversation

zddr commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 17, 2025

Uh oh!

zddr commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

zddr commented Jan 21, 2025

Uh oh!

zddr commented Jan 21, 2025

Uh oh!

zddr commented Jan 21, 2025

Uh oh!

zddr commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Uh oh!

zfr9527 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 24, 2025

Uh oh!

Jibing-Li left a comment

Choose a reason for hiding this comment

Uh oh!

zddr commented Feb 13, 2025

Uh oh!

github-actions Bot commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zddr commented Jan 17, 2025 •

edited

Loading