[fix](dynamic partition) drop partition exclude history_partition_num by yujun777 · Pull Request #37539 · apache/doris

yujun777 · 2024-07-09T08:45:28Z

FIX:

When dropping dynamic partition, PR #35778 will use math.max(start, -history_partition_num) as the first partition, but it may delete users' partitions if they specify both start and history_partition_num inappropriately. For safety reason, revert this behavious changed, only use start as the first partition when dropping partitions.

For those who had specified a very small start value, drop partitions will catch an exception , and stop dropping this table's partition and then record this error in dynamic info. Users can use command SHOW DYNAMIC PARTITION TABLES FROM DBXXX to know this error. From this error, it will give user hint to modify start if they really specify a error start.

doris-robot · 2024-07-09T08:45:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

yujun777 · 2024-07-09T08:45:39Z

run buildall

zclllyybb · 2024-07-09T08:54:17Z

need re-format FE

yujun777 · 2024-07-09T08:58:05Z

run buildall

doris-robot · 2024-07-09T09:32:51Z

TPC-H: Total hot run time: 39675 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10cc7a2322ce00d7cd5ce2d45625d276eeb6f074, data reload: false

------ Round 1 ----------------------------------
q1	17617	4342	4251	4251
q2	2019	199	187	187
q3	10450	1178	1100	1100
q4	10188	784	684	684
q5	7590	2639	2599	2599
q6	223	134	137	134
q7	965	591	597	591
q8	9228	2073	2076	2073
q9	8932	6496	6476	6476
q10	9000	3762	3743	3743
q11	453	243	235	235
q12	559	238	231	231
q13	17924	2998	2980	2980
q14	272	242	229	229
q15	529	471	487	471
q16	503	370	377	370
q17	971	670	674	670
q18	8025	7378	7405	7378
q19	4121	1581	1370	1370
q20	674	326	329	326
q21	5104	3232	4006	3232
q22	398	350	345	345
Total cold run time: 115745 ms
Total hot run time: 39675 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4416	4267	4267
q2	366	281	271	271
q3	3015	2828	2942	2828
q4	1964	1754	1712	1712
q5	5715	5578	5541	5541
q6	226	134	134	134
q7	2253	1889	1940	1889
q8	3289	3410	3451	3410
q9	8825	8836	8921	8836
q10	4116	3899	3791	3791
q11	615	506	512	506
q12	862	652	667	652
q13	16492	3209	3241	3209
q14	304	294	265	265
q15	564	492	483	483
q16	500	426	434	426
q17	1838	1530	1516	1516
q18	8107	8504	8525	8504
q19	4151	1692	1749	1692
q20	3676	1973	1920	1920
q21	6364	5153	4992	4992
q22	724	542	571	542
Total cold run time: 78349 ms
Total hot run time: 57386 ms

doris-robot · 2024-07-09T09:43:52Z

TPC-DS: Total hot run time: 174239 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10cc7a2322ce00d7cd5ce2d45625d276eeb6f074, data reload: false

query1	931	378	365	365
query2	6456	2494	2404	2404
query3	6630	219	228	219
query4	21309	17428	17216	17216
query5	3638	475	475	475
query6	257	172	194	172
query7	4595	308	299	299
query8	329	302	300	300
query9	8718	2454	2423	2423
query10	434	280	262	262
query11	10640	10144	10089	10089
query12	119	86	91	86
query13	1644	380	372	372
query14	10558	7613	7659	7613
query15	279	187	194	187
query16	7798	313	313	313
query17	1865	560	513	513
query18	1368	274	275	274
query19	320	157	147	147
query20	90	82	107	82
query21	201	126	121	121
query22	4520	4212	3988	3988
query23	33648	33542	33697	33542
query24	10186	2883	2857	2857
query25	564	402	385	385
query26	696	151	149	149
query27	2092	285	277	277
query28	6172	2128	2139	2128
query29	891	656	629	629
query30	258	157	159	157
query31	937	745	772	745
query32	98	55	56	55
query33	661	327	296	296
query34	886	510	486	486
query35	679	557	563	557
query36	1108	977	991	977
query37	149	88	92	88
query38	2960	2853	2849	2849
query39	937	861	831	831
query40	215	127	123	123
query41	51	54	53	53
query42	109	101	105	101
query43	595	567	572	567
query44	1083	726	716	716
query45	189	161	155	155
query46	1073	721	743	721
query47	1845	1803	1760	1760
query48	370	294	306	294
query49	825	402	429	402
query50	780	397	414	397
query51	6803	6796	6765	6765
query52	108	96	99	96
query53	368	302	301	301
query54	910	448	439	439
query55	76	74	73	73
query56	293	267	264	264
query57	1114	1024	1064	1024
query58	249	248	244	244
query59	3430	3489	3178	3178
query60	342	275	284	275
query61	96	91	93	91
query62	781	654	630	630
query63	316	292	297	292
query64	9130	2191	1618	1618
query65	3131	3114	3137	3114
query66	758	328	327	327
query67	15543	15151	14870	14870
query68	4439	537	540	537
query69	523	424	362	362
query70	1168	1114	1131	1114
query71	388	291	289	289
query72	7007	5442	5620	5442
query73	744	335	332	332
query74	5877	5526	5475	5475
query75	3381	2646	2704	2646
query76	2568	879	946	879
query77	449	368	306	306
query78	9515	8894	8837	8837
query79	3044	513	519	513
query80	2092	481	476	476
query81	595	225	230	225
query82	933	133	137	133
query83	287	179	174	174
query84	286	95	86	86
query85	1965	319	305	305
query86	489	292	336	292
query87	3307	3129	3088	3088
query88	4341	2440	2441	2440
query89	480	396	386	386
query90	1768	187	184	184
query91	133	119	112	112
query92	61	50	52	50
query93	2712	510	500	500
query94	1175	210	213	210
query95	409	322	327	322
query96	603	274	267	267
query97	3199	3012	3013	3012
query98	294	202	205	202
query99	1555	1233	1225	1225
Total cold run time: 271991 ms
Total hot run time: 174239 ms

doris-robot · 2024-07-09T09:49:24Z

ClickBench: Total hot run time: 30.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10cc7a2322ce00d7cd5ce2d45625d276eeb6f074, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.07	0.08
query5	0.50	0.47	0.49
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.47	0.50
query10	0.53	0.53	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.77	0.78	0.77
query15	0.85	0.80	0.81
query16	0.37	0.37	0.36
query17	0.99	0.99	0.96
query18	0.24	0.22	0.22
query19	1.79	1.66	1.66
query20	0.01	0.01	0.01
query21	15.38	0.78	0.66
query22	4.14	7.47	1.72
query23	18.29	1.39	1.28
query24	2.16	0.22	0.21
query25	0.17	0.09	0.09
query26	0.28	0.21	0.21
query27	0.46	0.25	0.23
query28	13.19	1.03	1.02
query29	12.68	3.29	3.29
query30	0.26	0.05	0.05
query31	2.87	0.40	0.38
query32	3.27	0.48	0.46
query33	2.86	2.91	2.98
query34	17.02	4.35	4.32
query35	4.39	4.44	4.43
query36	0.66	0.46	0.50
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.13	0.12
query41	0.09	0.04	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 30.34 s

yujun777 · 2024-07-09T10:16:59Z

run external

yujun777 · 2024-07-09T10:17:04Z

run p1

yujun777 · 2024-07-09T10:17:17Z

run cloud_p1

yujun777 · 2024-07-09T10:22:05Z

run p0

yujun777 · 2024-07-09T10:47:19Z

run p0

deardeng

LGTM

github-actions · 2024-07-09T10:56:38Z

PR approved by anyone and no changes requested.

yujun777 · 2024-07-09T11:13:04Z

run p1

yujun777 · 2024-07-09T11:13:11Z

run cloud_p1

yujun777 · 2024-07-09T11:15:19Z

run external

dataroaring

LGTM

github-actions · 2024-07-09T12:57:07Z

PR approved by at least one committer and no changes requested.

dataroaring · 2024-07-09T13:14:19Z

run buildall

…ion_num

dataroaring · 2024-07-09T13:18:54Z

run buildall

doris-robot · 2024-07-09T13:59:41Z

TPC-H: Total hot run time: 39415 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86af73633b5068d48660af6ccb30c98b6f364591, data reload: false

------ Round 1 ----------------------------------
q1	17626	4711	4228	4228
q2	2010	192	185	185
q3	10455	1216	1017	1017
q4	10189	717	789	717
q5	7499	2621	2579	2579
q6	217	137	134	134
q7	947	588	594	588
q8	9225	2069	2044	2044
q9	8869	6467	6433	6433
q10	8994	3732	3709	3709
q11	473	238	238	238
q12	486	228	223	223
q13	17779	2962	3008	2962
q14	249	229	225	225
q15	530	485	501	485
q16	500	388	376	376
q17	950	626	660	626
q18	7922	7461	7324	7324
q19	5418	1438	1489	1438
q20	659	331	329	329
q21	4989	3222	3830	3222
q22	393	333	335	333
Total cold run time: 116379 ms
Total hot run time: 39415 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4345	4180	4210	4180
q2	368	273	268	268
q3	2932	2719	2866	2719
q4	1972	1730	1694	1694
q5	5596	5506	5460	5460
q6	236	136	130	130
q7	2145	1827	1894	1827
q8	3230	3389	3345	3345
q9	8652	8642	8818	8642
q10	4063	3884	3681	3681
q11	606	489	490	489
q12	822	648	670	648
q13	16984	3145	3197	3145
q14	305	288	276	276
q15	511	475	478	475
q16	479	433	442	433
q17	1791	1503	1474	1474
q18	8032	7737	7805	7737
q19	1792	1662	1536	1536
q20	2132	1836	1864	1836
q21	5009	4897	4684	4684
q22	625	512	592	512
Total cold run time: 72627 ms
Total hot run time: 55191 ms

doris-robot · 2024-07-09T14:10:51Z

TPC-DS: Total hot run time: 173448 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86af73633b5068d48660af6ccb30c98b6f364591, data reload: false

query1	903	370	367	367
query2	6458	2382	2390	2382
query3	6648	204	216	204
query4	28160	17372	17231	17231
query5	3754	473	482	473
query6	264	169	159	159
query7	4591	280	283	280
query8	327	288	271	271
query9	8665	2381	2367	2367
query10	450	274	276	274
query11	11637	9990	9787	9787
query12	117	89	82	82
query13	1648	390	392	390
query14	10313	7924	7752	7752
query15	243	186	196	186
query16	7674	326	321	321
query17	1781	552	539	539
query18	1840	289	284	284
query19	197	155	157	155
query20	91	88	81	81
query21	202	128	121	121
query22	4271	4095	3982	3982
query23	33916	33637	33553	33553
query24	10863	2879	2880	2879
query25	606	435	411	411
query26	705	155	155	155
query27	2288	281	280	280
query28	5918	2118	2113	2113
query29	907	645	655	645
query30	246	155	153	153
query31	987	760	795	760
query32	92	55	53	53
query33	722	299	312	299
query34	902	486	507	486
query35	669	551	576	551
query36	1104	966	988	966
query37	148	86	86	86
query38	3016	2789	2803	2789
query39	892	855	806	806
query40	204	120	118	118
query41	56	49	49	49
query42	118	110	103	103
query43	575	548	548	548
query44	1138	756	740	740
query45	193	159	158	158
query46	1071	703	717	703
query47	1844	1760	1760	1760
query48	387	301	309	301
query49	848	408	415	408
query50	787	390	400	390
query51	6774	6790	6745	6745
query52	104	89	93	89
query53	356	289	292	289
query54	842	454	444	444
query55	74	74	72	72
query56	308	263	276	263
query57	1153	1057	1073	1057
query58	247	243	260	243
query59	3413	3371	3009	3009
query60	301	286	279	279
query61	100	118	92	92
query62	795	621	643	621
query63	322	292	286	286
query64	9150	2171	1628	1628
query65	3141	3089	3109	3089
query66	709	352	328	328
query67	15561	15105	14830	14830
query68	4578	544	537	537
query69	617	404	345	345
query70	1219	1152	1127	1127
query71	435	281	290	281
query72	8056	5471	5098	5098
query73	742	326	323	323
query74	5828	5502	5495	5495
query75	3491	2655	2615	2615
query76	2685	936	978	936
query77	660	303	315	303
query78	10223	8977	8862	8862
query79	3010	520	517	517
query80	1556	471	473	471
query81	591	221	219	219
query82	903	142	134	134
query83	339	164	168	164
query84	278	87	87	87
query85	1783	370	301	301
query86	460	322	334	322
query87	3299	3074	3058	3058
query88	4426	2453	2446	2446
query89	495	392	399	392
query90	1851	191	183	183
query91	131	103	101	101
query92	57	51	48	48
query93	4962	518	515	515
query94	1227	210	211	210
query95	401	320	317	317
query96	616	282	273	273
query97	3262	3016	3011	3011
query98	240	220	199	199
query99	1566	1243	1249	1243
Total cold run time: 284773 ms
Total hot run time: 173448 ms

dataroaring

LGTM

github-actions · 2024-07-09T14:14:22Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-07-09T14:16:24Z

ClickBench: Total hot run time: 30.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86af73633b5068d48660af6ccb30c98b6f364591, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.49	0.49	0.47
query6	1.12	0.74	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.49	0.49
query10	0.54	0.53	0.55
query11	0.16	0.11	0.12
query12	0.15	0.13	0.13
query13	0.59	0.59	0.58
query14	0.75	0.77	0.80
query15	0.86	0.81	0.82
query16	0.36	0.34	0.38
query17	0.96	1.05	0.93
query18	0.22	0.22	0.22
query19	1.83	1.72	1.70
query20	0.01	0.01	0.01
query21	15.40	0.77	0.67
query22	4.52	7.32	1.78
query23	18.27	1.41	1.26
query24	2.17	0.22	0.22
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.22	0.23
query28	13.23	1.02	1.00
query29	12.62	3.28	3.32
query30	0.25	0.06	0.06
query31	2.88	0.37	0.40
query32	3.28	0.48	0.47
query33	2.88	2.92	2.92
query34	17.07	4.38	4.34
query35	4.44	4.40	4.43
query36	0.66	0.46	0.47
query37	0.19	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.15	0.13	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110 s
Total hot run time: 30.38 s

…#37539) FIX: When dropping dynamic partition, PR #35778 will use math.max(start, -history_partition_num) as the first partition, but it may delete users' partitions if they specify both start and history_partition_num inappropriately. For safety reason, revert this behavious changed, only use start as the first partition when dropping partitions. For those who had specified a very small start value, drop partitions will catch an exception , and stop dropping this table's partition and then record this error in dynamic info. Users can use command `SHOW DYNAMIC PARTITION TABLES FROM DBXXX` to know this error. From this error, it will give user hint to modify start if they really specify a error start. --------- Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

…apache#37539) FIX: When dropping dynamic partition, PR apache#35778 will use math.max(start, -history_partition_num) as the first partition, but it may delete users' partitions if they specify both start and history_partition_num inappropriately. For safety reason, revert this behavious changed, only use start as the first partition when dropping partitions. For those who had specified a very small start value, drop partitions will catch an exception , and stop dropping this table's partition and then record this error in dynamic info. Users can use command `SHOW DYNAMIC PARTITION TABLES FROM DBXXX` to know this error. From this error, it will give user hint to modify start if they really specify a error start. --------- Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

…37539 (#37571)

…37539 (#37570) cherry pick from #37539 --------- Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

…pache#37539 (apache#37571)

zxealous · 2024-09-06T08:15:51Z

                    addPartitionClauses = getAddPartitionClause(db, olapTable, partitionColumn, partitionFormat,
                            executeFirstTime);
                }
+                clearDropPartitionFailedMsg(olapTable.getId());


why call clearDropPartitionFailedMsg() here?

This will cause the status to remain NORMAL even if add partition failed.

dataroaring added dev/2.1.x dev/2.0.x labels Jul 9, 2024

deardeng approved these changes Jul 9, 2024

View reviewed changes

github-actions Bot added the reviewed label Jul 9, 2024

dataroaring previously approved these changes Jul 9, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2024

dataroaring dismissed their stale review via 862d9c7 July 9, 2024 13:14

github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Jul 9, 2024

yujun777 and others added 3 commits July 9, 2024 21:18

drop dynamic partition only considering start, exclude history_partit…

9098b93

…ion_num

udpate

a2fd31e

Update test_dynamic_partition_failed.groovy

86af736

dataroaring force-pushed the fix-dynamic-partition-drop-partition branch from 862d9c7 to 86af736 Compare July 9, 2024 13:18

dataroaring approved these changes Jul 9, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2024

morningman merged commit bc8e660 into apache:master Jul 9, 2024

yujun777 mentioned this pull request Jul 10, 2024

[fix](dynamic partition) drop partition exclude history_partition_num #37539 #37570

Merged

yujun777 mentioned this pull request Jul 10, 2024

[fix](dynamic partition) drop partition exclude history_partition_num #37539 #37571

Merged

xiaokang pushed a commit that referenced this pull request Jul 10, 2024

[fix](dynamic partition) drop partition exclude history_partition_num #…

9ca846a

…37539 (#37571)

xiaokang added dev/2.0.13-merged and removed dev/2.0.x labels Jul 10, 2024

dataroaring added a commit that referenced this pull request Jul 10, 2024

[fix](dynamic partition) drop partition exclude history_partition_num #…

f58032f

…37539 (#37570) cherry pick from #37539 --------- Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 10, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](dynamic partition) drop partition exclude history_partition_num a…

4352d79

…pache#37539 (apache#37571)

zxealous reviewed Sep 6, 2024

View reviewed changes

Conversation

yujun777 commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

zclllyybb commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

deardeng left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

yujun777 commented Jul 9, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 9, 2024

Uh oh!

dataroaring commented Jul 9, 2024

Uh oh!

dataroaring commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

zxealous Sep 6, 2024

Choose a reason for hiding this comment

Uh oh!

zxealous Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

yujun777 commented Jul 9, 2024 •

edited

Loading

zxealous Sep 6, 2024 •

edited

Loading