Skip to content

[feature](restore) Support clean_tables/clean_partitions properties for restore job #39028#39366

Merged
w41ter merged 1 commit intoapache:branch-2.0from
w41ter:pick_39028_to_origin_branch-2.0
Aug 15, 2024
Merged

[feature](restore) Support clean_tables/clean_partitions properties for restore job #39028#39366
w41ter merged 1 commit intoapache:branch-2.0from
w41ter:pick_39028_to_origin_branch-2.0

Conversation

@w41ter
Copy link
Copy Markdown
Contributor

@w41ter w41ter commented Aug 14, 2024

cherry pick from #39028

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 14, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 14, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Aug 14, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 14, 2024

clang-tidy review says "All clean, LGTM! 👍"

@w41ter w41ter force-pushed the pick_39028_to_origin_branch-2.0 branch 2 times, most recently from f3c3674 to 35dbd4c Compare August 15, 2024 02:10
@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 15, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

TPC-H: Total hot run time: 49972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35dbd4cce4f0ce36fee3fdf56b4bcbfb07ff3134, data reload: false

------ Round 1 ----------------------------------
q1	18302	4446	4397	4397
q2	2089	156	146	146
q3	10310	1862	1902	1862
q4	10660	1224	1369	1224
q5	8475	3905	3930	3905
q6	232	126	127	126
q7	2098	1614	1651	1614
q8	9522	2777	2743	2743
q9	10915	10564	10756	10564
q10	11918	3757	3531	3531
q11	430	248	234	234
q12	470	300	303	300
q13	18864	4001	4050	4001
q14	368	325	330	325
q15	510	461	462	461
q16	687	591	587	587
q17	1135	998	958	958
q18	7282	6918	6730	6730
q19	1725	1576	1488	1488
q20	519	325	292	292
q21	4427	4089	4092	4089
q22	489	399	395	395
Total cold run time: 121427 ms
Total hot run time: 49972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4313	4305	4305
q2	322	225	220	220
q3	4153	4120	4135	4120
q4	2745	2758	2763	2758
q5	7189	7093	7096	7093
q6	237	117	118	117
q7	3229	2854	2850	2850
q8	4355	4431	4511	4431
q9	16953	16729	16685	16685
q10	4392	4349	4373	4349
q11	758	694	712	694
q12	1037	853	848	848
q13	7181	3749	3725	3725
q14	451	427	418	418
q15	497	459	465	459
q16	730	678	700	678
q17	3833	3848	3925	3848
q18	8800	8746	8725	8725
q19	1704	1671	1647	1647
q20	2364	2103	2081	2081
q21	8569	8454	8426	8426
q22	1057	918	949	918
Total cold run time: 84910 ms
Total hot run time: 79395 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

TPC-DS: Total hot run time: 202060 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35dbd4cce4f0ce36fee3fdf56b4bcbfb07ff3134, data reload: false

query1	928	394	419	394
query2	6557	2302	2237	2237
query3	6920	207	199	199
query4	21569	18048	18060	18048
query5	19728	6490	6467	6467
query6	282	221	223	221
query7	4165	302	313	302
query8	285	250	241	241
query9	3160	2739	2663	2663
query10	436	297	289	289
query11	11475	10829	10778	10778
query12	122	81	74	74
query13	5593	645	660	645
query14	18120	13202	13422	13202
query15	380	230	225	225
query16	6462	286	263	263
query17	1729	1453	890	890
query18	2296	424	413	413
query19	202	154	150	150
query20	79	79	77	77
query21	188	94	98	94
query22	5312	5073	5185	5073
query23	32461	31642	31855	31642
query24	6893	6440	6528	6440
query25	535	433	433	433
query26	527	161	157	157
query27	1887	288	291	288
query28	6174	2321	2271	2271
query29	2991	2685	2881	2685
query30	240	166	168	166
query31	907	730	773	730
query32	68	62	60	60
query33	404	259	278	259
query34	841	464	481	464
query35	1154	913	938	913
query36	1234	1288	1295	1288
query37	89	61	61	61
query38	3054	2921	2925	2921
query39	1362	1324	1318	1318
query40	205	95	96	95
query41	39	38	37	37
query42	86	82	79	79
query43	608	602	599	599
query44	1125	712	720	712
query45	242	227	226	226
query46	1233	936	975	936
query47	1831	1659	1765	1659
query48	1001	676	662	662
query49	622	376	374	374
query50	877	626	606	606
query51	4802	4709	4717	4709
query52	85	89	85	85
query53	444	340	323	323
query54	2662	2476	2468	2468
query55	83	83	78	78
query56	235	235	211	211
query57	1104	1043	1186	1043
query58	223	189	209	189
query59	3494	3441	3296	3296
query60	212	207	218	207
query61	97	94	95	94
query62	825	523	476	476
query63	501	344	338	338
query64	2382	1550	1427	1427
query65	3696	3558	3581	3558
query66	850	379	379	379
query67	15300	15028	14783	14783
query68	10554	655	657	655
query69	588	354	356	354
query70	2012	1632	1524	1524
query71	434	309	307	307
query72	6563	3512	3496	3496
query73	752	315	315	315
query74	6371	5832	5813	5813
query75	5497	3747	3678	3678
query76	6958	1172	1202	1172
query77	1194	251	260	251
query78	12620	12047	12197	12047
query79	9642	661	664	661
query80	857	397	399	397
query81	483	234	238	234
query82	800	96	98	96
query83	171	139	133	133
query84	258	73	69	69
query85	873	329	314	314
query86	330	310	339	310
query87	3229	3032	3047	3032
query88	4389	2320	2327	2320
query89	482	272	293	272
query90	1978	210	217	210
query91	163	119	122	119
query92	60	51	57	51
query93	7053	601	616	601
query94	696	208	206	206
query95	1150	1052	1040	1040
query96	647	334	323	323
query97	6404	6323	6400	6323
query98	198	178	171	171
query99	2918	890	876	876
Total cold run time: 317997 ms
Total hot run time: 202060 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35dbd4cce4f0ce36fee3fdf56b4bcbfb07ff3134, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.24	0.04	0.05
query4	1.77	0.06	0.07
query5	0.53	0.52	0.52
query6	1.29	0.61	0.67
query7	0.01	0.00	0.01
query8	0.03	0.02	0.02
query9	0.52	0.51	0.48
query10	0.54	0.54	0.55
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.61	0.61	0.61
query14	0.78	0.79	0.77
query15	0.77	0.75	0.76
query16	0.36	0.38	0.37
query17	0.99	1.02	1.03
query18	0.23	0.26	0.23
query19	1.91	1.86	1.85
query20	0.01	0.02	0.02
query21	15.46	0.55	0.55
query22	2.06	1.71	1.80
query23	17.12	1.16	0.84
query24	6.32	1.06	1.26
query25	0.36	0.14	0.05
query26	0.68	0.14	0.15
query27	0.04	0.04	0.03
query28	6.47	0.72	0.76
query29	12.69	2.31	2.12
query30	0.57	0.51	0.54
query31	2.80	0.39	0.37
query32	3.37	0.51	0.50
query33	3.08	3.10	3.11
query34	15.25	4.81	4.80
query35	4.82	4.80	4.85
query36	1.08	1.04	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.48 s
Total hot run time: 30.68 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 35dbd4cce4f0ce36fee3fdf56b4bcbfb07ff3134 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@w41ter
[feature](restore) Support clean_tables/clean_partitions properties f…
1e11d48 
…or restore job (apache#39028)

The restore will keep the existing tables/partitions in which the
restore target is not contained, this PR adds a property clean_restore
to indicate that the restore job needs to recycle those
tables/partitions

The CCR part PR is: selectdb/ccr-syncer#128.
The website PR is: apache/doris-website#999
@w41ter w41ter force-pushed the pick_39028_to_origin_branch-2.0 branch from 35dbd4c to 1e11d48 Compare August 15, 2024 03:13
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Aug 15, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

TPC-H: Total hot run time: 49464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e11d482834b569be5ca991fba02eb968d3267b9, data reload: false

------ Round 1 ----------------------------------
q1	18251	4369	4312	4312
q2	2077	163	151	151
q3	10291	1897	1945	1897
q4	10103	1266	1335	1266
q5	8421	3882	3925	3882
q6	239	125	124	124
q7	2032	1594	1584	1584
q8	9538	2732	2703	2703
q9	13649	10277	10164	10164
q10	8653	3497	3534	3497
q11	412	252	256	252
q12	471	304	306	304
q13	18334	3941	4054	3941
q14	351	332	326	326
q15	507	463	442	442
q16	675	584	571	571
q17	1124	912	962	912
q18	7322	6873	6987	6873
q19	1673	1552	1543	1543
q20	547	286	320	286
q21	4395	4096	4039	4039
q22	495	395	398	395
Total cold run time: 119560 ms
Total hot run time: 49464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4347	4352	4302	4302
q2	321	225	230	225
q3	4182	4125	4156	4125
q4	2753	2761	2758	2758
q5	7166	7152	7110	7110
q6	243	122	124	122
q7	3232	2803	2811	2803
q8	4358	4482	4505	4482
q9	16903	16820	16839	16820
q10	4198	4213	4236	4213
q11	766	711	689	689
q12	1024	844	867	844
q13	6920	3755	3721	3721
q14	477	439	436	436
q15	511	456	467	456
q16	737	693	682	682
q17	3794	3946	3900	3900
q18	8843	8749	8807	8749
q19	1717	1640	1663	1640
q20	2371	2187	2140	2140
q21	8463	8397	8509	8397
q22	1069	953	968	953
Total cold run time: 84395 ms
Total hot run time: 79567 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

TPC-DS: Total hot run time: 204435 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e11d482834b569be5ca991fba02eb968d3267b9, data reload: false

query1	947	391	436	391
query2	6536	2239	2119	2119
query3	6922	205	200	200
query4	21509	18042	18038	18038
query5	19747	6510	6513	6510
query6	289	220	231	220
query7	4157	308	314	308
query8	256	253	241	241
query9	3187	2761	2688	2688
query10	407	306	298	298
query11	11280	10647	10693	10647
query12	125	75	76	75
query13	5590	664	656	656
query14	17913	13474	13240	13240
query15	356	227	235	227
query16	6463	281	258	258
query17	1741	1450	882	882
query18	2313	415	427	415
query19	211	154	153	153
query20	81	79	82	79
query21	189	99	100	99
query22	5259	4987	4966	4966
query23	32584	31966	31933	31933
query24	6931	6540	6478	6478
query25	538	433	424	424
query26	532	161	160	160
query27	1844	308	306	306
query28	5972	2324	2300	2300
query29	2865	2780	2702	2702
query30	246	167	168	167
query31	908	734	771	734
query32	72	64	64	64
query33	404	259	258	258
query34	851	511	506	506
query35	1126	989	876	876
query36	1555	1276	1144	1144
query37	91	60	60	60
query38	3125	2934	2988	2934
query39	1360	1318	1325	1318
query40	203	94	95	94
query41	39	37	38	37
query42	89	86	93	86
query43	638	591	597	591
query44	1151	717	719	717
query45	248	225	232	225
query46	1247	987	980	980
query47	1836	1725	1968	1725
query48	974	685	685	685
query49	631	360	368	360
query50	870	641	588	588
query51	4787	4616	4727	4616
query52	101	85	93	85
query53	441	326	316	316
query54	2654	2476	2448	2448
query55	83	91	78	78
query56	223	221	208	208
query57	1145	1194	1051	1051
query58	226	209	205	205
query59	3622	3402	3264	3264
query60	218	193	199	193
query61	97	99	98	98
query62	839	483	438	438
query63	487	344	341	341
query64	2508	1575	1473	1473
query65	3632	3577	3565	3565
query66	770	379	402	379
query67	16416	16003	15559	15559
query68	9790	677	681	677
query69	600	343	344	343
query70	1749	1380	1496	1380
query71	418	318	315	315
query72	6518	3484	3477	3477
query73	737	322	321	321
query74	6300	5834	5828	5828
query75	5282	3673	3713	3673
query76	6297	1157	1221	1157
query77	1064	252	265	252
query78	12598	13764	14251	13764
query79	11423	644	642	642
query80	858	407	400	400
query81	498	239	235	235
query82	261	96	97	96
query83	192	132	136	132
query84	272	72	71	71
query85	760	319	319	319
query86	347	292	300	292
query87	3249	3101	3078	3078
query88	4601	2323	2306	2306
query89	359	289	301	289
query90	2641	208	203	203
query91	160	129	121	121
query92	57	57	52	52
query93	1119	581	590	581
query94	1199	216	214	214
query95	1138	1107	1052	1052
query96	632	333	328	328
query97	6432	6313	6420	6313
query98	191	180	170	170
query99	2992	903	875	875
Total cold run time: 313391 ms
Total hot run time: 204435 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

ClickBench: Total hot run time: 31.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e11d482834b569be5ca991fba02eb968d3267b9, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.77	0.07	0.07
query5	0.54	0.53	0.52
query6	1.26	0.62	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.51	0.48	0.47
query10	0.54	0.53	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.10
query13	0.61	0.62	0.62
query14	0.80	0.80	0.77
query15	0.78	0.75	0.76
query16	0.38	0.41	0.36
query17	1.00	1.02	1.00
query18	0.21	0.26	0.24
query19	1.89	1.82	1.86
query20	0.02	0.01	0.01
query21	15.48	0.55	0.55
query22	2.15	2.62	2.07
query23	17.64	1.02	1.09
query24	6.30	1.25	1.36
query25	0.37	0.11	0.05
query26	0.71	0.16	0.16
query27	0.05	0.04	0.04
query28	6.19	0.74	0.75
query29	12.59	2.34	2.29
query30	0.61	0.53	0.53
query31	2.80	0.38	0.38
query32	3.40	0.50	0.50
query33	3.11	3.10	3.03
query34	15.26	4.81	4.79
query35	4.83	4.83	4.85
query36	1.08	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 103.91 s
Total hot run time: 31.56 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1e11d482834b569be5ca991fba02eb968d3267b9 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@w41ter w41ter merged commit 2168f09 into apache:branch-2.0 Aug 15, 2024
@w41ter w41ter deleted the pick_39028_to_origin_branch-2.0 branch August 15, 2024 06:16
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@w41ter @Baymine
[feature](restore) Support clean_tables/clean_partitions properties f…
7342cf8 
…or restore job apache#39028 (apache#39366)

cherry pick from apache#39028
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/planner Issues or PRs related to the query planner kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@w41ter @doris-robot