Skip to content

[opt](parse) optimize parsing string to datetime#38385

Merged
HappenLee merged 1 commit intoapache:masterfrom
zclllyybb:opt_datetime
Jul 29, 2024
Merged

[opt](parse) optimize parsing string to datetime#38385
HappenLee merged 1 commit intoapache:masterfrom
zclllyybb:opt_datetime

Conversation

@zclllyybb
Copy link
Copy Markdown
Contributor

@zclllyybb zclllyybb commented Jul 25, 2024

Proposed changes

Issue Number: close #xxx

create table str(
k0 varchar not null
)
DISTRIBUTED BY HASH(`k0`) BUCKETS auto
properties("replication_num" = "1");

insert into str select '2034-12-15 12:12:12' from numbers("number" = "50000000");
set parallel_pipeline_task_num=1;

--- origin
mysql [test]>select count(cast(k0 as datetime)) from str;
+----------------------------------+
| count(cast(k0 as DATETIMEV2(0))) |
+----------------------------------+
|                         50000000 |
+----------------------------------+
1 row in set (7.09 sec)

--- opt function and isspace
mysql [test]>select count(cast(k0 as datetime)) from str;
+----------------------------------+
| count(cast(k0 as DATETIMEV2(0))) |
+----------------------------------+
|                         50000000 |
+----------------------------------+
1 row in set (5.28 sec)

--- and ispunct
mysql [test]>select count(cast(k0 as datetime)) from str;
+----------------------------------+
| count(cast(k0 as DATETIMEV2(0))) |
+----------------------------------+
|                         50000000 |
+----------------------------------+
1 row in set (5.14 sec)

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Copy Markdown
Contributor Author

zclllyybb commented Jul 25, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 25, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment thread be/src/vec/runtime/vdatetime_value.cpp
Comment thread be/src/vec/runtime/vdatetime_value.cpp
@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

TPC-H: Total hot run time: 39179 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b19d38cedc9d613ab16935a84ac761641920dfb, data reload: false

------ Round 1 ----------------------------------
q1	17644	4367	4254	4254
q2	2031	195	197	195
q3	10451	1232	1145	1145
q4	10133	650	723	650
q5	7601	2663	2665	2663
q6	218	137	139	137
q7	941	592	587	587
q8	9238	1879	1893	1879
q9	8781	6538	6561	6538
q10	8783	3796	3765	3765
q11	473	245	251	245
q12	520	225	222	222
q13	18706	2972	2985	2972
q14	284	235	234	234
q15	515	499	468	468
q16	494	381	384	381
q17	948	658	712	658
q18	7984	7505	7347	7347
q19	6891	1014	973	973
q20	700	331	330	330
q21	5164	3244	3960	3244
q22	350	293	292	292
Total cold run time: 118850 ms
Total hot run time: 39179 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4481	4201	4265	4201
q2	375	267	266	266
q3	2964	2873	2890	2873
q4	1969	1693	1738	1693
q5	5642	5542	5457	5457
q6	225	141	131	131
q7	2145	1844	1827	1827
q8	3242	3408	3441	3408
q9	8756	8754	8779	8754
q10	4079	3825	3778	3778
q11	585	510	514	510
q12	823	656	640	640
q13	15932	3153	3138	3138
q14	304	293	284	284
q15	535	487	505	487
q16	478	449	424	424
q17	1813	1542	1520	1520
q18	8137	7931	7932	7931
q19	1736	1662	1534	1534
q20	2432	1895	1848	1848
q21	5070	4867	4617	4617
q22	574	528	504	504
Total cold run time: 72297 ms
Total hot run time: 55825 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

TPC-DS: Total hot run time: 173235 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b19d38cedc9d613ab16935a84ac761641920dfb, data reload: false

query1	901	362	365	362
query2	6428	1892	1866	1866
query3	6628	203	214	203
query4	27565	17428	17311	17311
query5	3781	478	468	468
query6	268	174	163	163
query7	4573	286	288	286
query8	241	185	191	185
query9	8619	2432	2422	2422
query10	433	280	269	269
query11	10599	10096	10279	10096
query12	120	87	86	86
query13	1640	398	372	372
query14	10136	7653	7610	7610
query15	225	168	171	168
query16	7587	503	454	454
query17	1419	566	549	549
query18	1928	325	282	282
query19	189	141	138	138
query20	95	84	84	84
query21	207	95	97	95
query22	4265	4038	4049	4038
query23	34051	33591	33490	33490
query24	11132	2864	2915	2864
query25	620	390	379	379
query26	1089	154	149	149
query27	2263	280	281	280
query28	6022	2084	2074	2074
query29	880	450	440	440
query30	260	158	170	158
query31	983	765	783	765
query32	95	52	55	52
query33	769	325	324	324
query34	946	473	493	473
query35	902	733	752	733
query36	1143	994	994	994
query37	136	81	85	81
query38	3024	2844	2802	2802
query39	916	857	846	846
query40	198	112	114	112
query41	45	43	44	43
query42	118	101	105	101
query43	509	472	453	453
query44	1184	732	730	730
query45	206	177	181	177
query46	1082	756	757	756
query47	1846	1777	1757	1757
query48	374	291	293	291
query49	844	406	417	406
query50	785	394	401	394
query51	6749	6643	6801	6643
query52	102	87	89	87
query53	255	183	179	179
query54	874	435	447	435
query55	74	72	72	72
query56	286	295	274	274
query57	1138	1054	1035	1035
query58	276	257	259	257
query59	2803	2753	2745	2745
query60	308	281	305	281
query61	94	93	97	93
query62	801	650	661	650
query63	211	183	174	174
query64	9325	2274	1690	1690
query65	3171	3091	3088	3088
query66	754	326	329	326
query67	15326	15233	14878	14878
query68	4426	549	544	544
query69	529	355	308	308
query70	1091	1053	1049	1049
query71	374	268	275	268
query72	7002	5592	5602	5592
query73	746	327	336	327
query74	6232	5710	5620	5620
query75	3331	2673	2699	2673
query76	2208	952	900	900
query77	487	313	308	308
query78	9992	9058	8965	8965
query79	4099	525	538	525
query80	2017	491	493	491
query81	600	221	223	221
query82	1472	136	135	135
query83	338	176	177	176
query84	271	76	79	76
query85	1438	371	299	299
query86	468	301	310	301
query87	3298	3114	3111	3111
query88	4577	2381	2390	2381
query89	433	285	282	282
query90	1768	193	183	183
query91	125	98	98	98
query92	58	48	49	48
query93	5187	555	541	541
query94	879	283	298	283
query95	366	260	264	260
query96	608	272	267	267
query97	3217	3029	3000	3000
query98	219	193	194	193
query99	1603	1293	1272	1272
Total cold run time: 282275 ms
Total hot run time: 173235 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b19d38cedc9d613ab16935a84ac761641920dfb, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.07	0.06
query5	0.47	0.49	0.47
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.13	0.12
query13	0.60	0.60	0.58
query14	0.75	0.78	0.78
query15	0.84	0.80	0.80
query16	0.36	0.34	0.37
query17	0.99	0.94	1.04
query18	0.23	0.21	0.21
query19	1.80	1.78	1.79
query20	0.01	0.01	0.01
query21	15.42	0.78	0.66
query22	4.17	6.70	2.16
query23	18.29	1.37	1.24
query24	2.12	0.22	0.23
query25	0.16	0.09	0.09
query26	0.29	0.22	0.21
query27	0.45	0.23	0.23
query28	13.19	1.03	1.00
query29	12.64	3.30	3.26
query30	0.27	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.48	0.47
query33	2.91	2.90	2.88
query34	17.04	4.34	4.37
query35	4.43	4.36	4.42
query36	0.65	0.49	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.59 s
Total hot run time: 30.72 s

HappenLee
HappenLee approved these changes Jul 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 29, 2024

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit bb5b05b into apache:master Jul 29, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@zclllyybb @dataroaring
[opt](parse) optimize parsing string to datetime (#38385)
728178d
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@zclllyybb @dataroaring
[opt](parse) optimize parsing string to datetime (#38385)
65473e2
@zclllyybb zclllyybb deleted the opt_datetime branch August 23, 2024 09:54
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Aug 23, 2024
@zclllyybb
[opt](parse) optimize parsing string to datetime (apache#38385)
eafad7d
@zclllyybb zclllyybb mentioned this pull request Aug 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 24, 2024
@zclllyybb
[branch-2.1] pick some prs (#39860)
460605a 
## Proposed changes

Issue Number: close #xxx

#38385 optimize parsing datetime
#38978 make stream load failure
message more clear and disable some error's stacktrace by default
#39255 fix random function coredump
#39324 fix function corr
inconsistency with doc
#39449 check auto partitoin nullity
when creating partition
#39695 make
DynamicPartitionScheduler immediately know interval's change
#39754 Add some partition expr check
on creating table
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zclllyybb @doris-robot @BiteTheDDDDt @HappenLee @yiguolei @dataroaring