Skip to content

[Feat](nereids) support date function in partition prune#38743

Merged
morrySnow merged 1 commit intoapache:masterfrom
feiniaofeiafei:support_date_func_in_partition_prune
Aug 2, 2024
Merged

[Feat](nereids) support date function in partition prune#38743
morrySnow merged 1 commit intoapache:masterfrom
feiniaofeiafei:support_date_func_in_partition_prune

Conversation

@feiniaofeiafei
Copy link
Copy Markdown
Contributor

@feiniaofeiafei feiniaofeiafei commented Aug 1, 2024 •
edited
Loading

This pr supports to compute date(dt)>10. For example, there is a partition ('2020-01-01 10:00:00', '2020-02-05 10:00:00') ,partition column is dt, and a predicate date(dt)<'2019-01-01 ', this partition can be pruned.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 1, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei
Copy link
Copy Markdown
Contributor Author

feiniaofeiafei commented Aug 1, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 1, 2024

TPC-H: Total hot run time: 41737 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 794bc5d5d31d11229bbc32e51678f5a1907a0347, data reload: false

------ Round 1 ----------------------------------
q1	17601	4121	4048	4048
q2	2020	203	198	198
q3	11095	1371	1387	1371
q4	10761	799	891	799
q5	8240	3038	2999	2999
q6	226	142	141	141
q7	1059	627	627	627
q8	9970	1885	1961	1885
q9	8654	6700	6618	6618
q10	8741	3897	3862	3862
q11	428	249	253	249
q12	421	235	231	231
q13	17784	2981	2959	2959
q14	277	249	241	241
q15	533	485	494	485
q16	525	390	383	383
q17	953	903	898	898
q18	8004	7275	7170	7170
q19	1402	1209	1208	1208
q20	575	334	360	334
q21	5321	4762	4750	4750
q22	353	281	295	281
Total cold run time: 114943 ms
Total hot run time: 41737 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4053	4021	3993	3993
q2	330	225	223	223
q3	3007	3153	3123	3123
q4	2025	2016	2085	2016
q5	5517	5459	5392	5392
q6	218	132	140	132
q7	2144	1780	1849	1780
q8	3272	3327	3323	3323
q9	8714	8599	8482	8482
q10	3960	4104	4039	4039
q11	562	480	461	461
q12	782	609	602	602
q13	13413	3153	3103	3103
q14	299	284	278	278
q15	541	480	496	480
q16	461	417	412	412
q17	1766	1751	1751	1751
q18	8169	7815	7831	7815
q19	1709	1770	1705	1705
q20	2056	1832	1852	1832
q21	5840	5301	5572	5301
q22	550	492	472	472
Total cold run time: 69388 ms
Total hot run time: 56715 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 1, 2024

TPC-DS: Total hot run time: 170861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 794bc5d5d31d11229bbc32e51678f5a1907a0347, data reload: false

query1	924	383	374	374
query2	6314	1722	1721	1721
query3	6657	215	229	215
query4	19185	17452	17345	17345
query5	3696	527	525	525
query6	272	174	164	164
query7	4610	305	298	298
query8	250	197	197	197
query9	8499	2376	2349	2349
query10	431	281	288	281
query11	10551	10019	10067	10019
query12	126	94	88	88
query13	1619	386	368	368
query14	8425	7669	7672	7669
query15	207	163	162	162
query16	6965	457	444	444
query17	960	579	606	579
query18	1909	303	282	282
query19	188	146	146	146
query20	91	85	86	85
query21	204	109	102	102
query22	4392	4090	4124	4090
query23	33900	33545	33580	33545
query24	10197	3118	3072	3072
query25	691	427	432	427
query26	1750	155	155	155
query27	2905	284	282	282
query28	7404	2020	2003	2003
query29	1270	423	433	423
query30	235	152	155	152
query31	941	781	778	778
query32	98	53	58	53
query33	685	316	331	316
query34	923	518	493	493
query35	865	766	763	763
query36	1092	911	889	889
query37	293	81	87	81
query38	3002	2853	2846	2846
query39	906	862	816	816
query40	261	114	111	111
query41	48	43	45	43
query42	130	106	100	100
query43	468	418	423	418
query44	1181	728	725	725
query45	206	176	177	176
query46	1096	830	778	778
query47	1851	1748	1756	1748
query48	381	291	287	287
query49	936	439	430	430
query50	883	428	439	428
query51	6804	6771	6652	6652
query52	99	91	90	90
query53	254	183	180	180
query54	604	459	445	445
query55	75	74	76	74
query56	281	249	267	249
query57	1173	1080	1049	1049
query58	253	269	280	269
query59	2667	2320	2432	2320
query60	300	275	287	275
query61	107	97	98	97
query62	887	661	680	661
query63	220	184	179	179
query64	5570	1907	1884	1884
query65	3179	3091	3102	3091
query66	1315	334	331	331
query67	15600	14979	14917	14917
query68	4510	579	590	579
query69	493	302	301	301
query70	1068	1061	1014	1014
query71	430	273	282	273
query72	7051	2693	2514	2514
query73	771	339	331	331
query74	6010	5671	5678	5671
query75	3394	2749	2770	2749
query76	2613	1206	1282	1206
query77	417	310	310	310
query78	9469	9009	8892	8892
query79	1862	541	536	536
query80	1168	520	504	504
query81	575	232	232	232
query82	1127	134	137	134
query83	240	177	180	177
query84	267	83	84	83
query85	1289	311	303	303
query86	399	289	317	289
query87	3301	3076	3077	3076
query88	2935	2430	2479	2430
query89	386	300	293	293
query90	1783	194	197	194
query91	124	102	101	101
query92	65	51	51	51
query93	1512	620	632	620
query94	797	293	312	293
query95	381	273	266	266
query96	597	292	283	283
query97	3284	3063	3081	3063
query98	221	204	197	197
query99	1620	1301	1292	1292
Total cold run time: 260620 ms
Total hot run time: 170861 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 1, 2024

ClickBench: Total hot run time: 29.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 794bc5d5d31d11229bbc32e51678f5a1907a0347, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.05	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.49	0.48	0.49
query6	1.15	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.57	0.53	0.51
query10	0.56	0.55	0.58
query11	0.15	0.11	0.12
query12	0.16	0.13	0.13
query13	0.63	0.61	0.60
query14	0.79	0.81	0.78
query15	0.90	0.87	0.87
query16	0.36	0.35	0.36
query17	0.98	0.99	1.00
query18	0.22	0.21	0.21
query19	1.88	1.78	1.69
query20	0.01	0.01	0.00
query21	15.41	0.78	0.65
query22	4.27	7.71	1.22
query23	17.92	1.27	1.34
query24	2.26	0.22	0.22
query25	0.18	0.09	0.08
query26	0.33	0.21	0.21
query27	0.46	0.23	0.24
query28	13.16	1.02	0.98
query29	12.51	3.33	3.31
query30	0.26	0.05	0.05
query31	2.87	0.40	0.41
query32	3.23	0.48	0.49
query33	2.93	2.99	2.97
query34	15.43	4.27	4.23
query35	4.33	4.28	4.33
query36	0.68	0.48	0.48
query37	0.20	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.93 s
Total hot run time: 29.9 s

@feiniaofeiafei
Copy link
Copy Markdown
Contributor Author

feiniaofeiafei commented Aug 2, 2024

run external

@feiniaofeiafei
Copy link
Copy Markdown
Contributor Author

feiniaofeiafei commented Aug 2, 2024

run cloud_p1

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 2, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 2, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit da226b9 into apache:master Aug 2, 2024
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Aug 5, 2024
@feiniaofeiafei
[Feat](nereids) support date function in partition prune (apache#38743)
394fec7 
This pr supports to compute date(dt)>10.
For example, there is a partition ('2020-01-01 10:00:00', '2020-02-05 10:00:00') ,
partition column is dt, and a predicate date(dt)<'2019-01-01 ',
this partition can be pruned.
@feiniaofeiafei feiniaofeiafei mentioned this pull request Aug 5, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 6, 2024
@feiniaofeiafei
[Feat](nereids) support date function in partition prune (#38743) (#3…
ab3057b 
…8898)

cherry-pick #38743 to branch-2.1
gavinchou pushed a commit that referenced this pull request Aug 8, 2024
@feiniaofeiafei @gavinchou
[Feat](nereids) support date function in partition prune (#38743)
0c1d935 
This pr supports to compute date(dt)>10.
For example, there is a partition ('2020-01-01 10:00:00', '2020-02-05 10:00:00') ,
partition column is dt, and a predicate date(dt)<'2019-01-01 ',
this partition can be pruned.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@feiniaofeiafei @doris-robot @924060929 @morrySnow @gavinchou @yiguolei