Skip to content

[fix](stream_load) fix stream load may failed caused by column name with keyword#35822

Merged
caiconghui merged 1 commit intomasterfrom
stream_load_error
Jun 4, 2024
Merged

[fix](stream_load) fix stream load may failed caused by column name with keyword#35822
caiconghui merged 1 commit intomasterfrom
stream_load_error

Conversation

@caiconghui
Copy link
Copy Markdown
Contributor

@caiconghui caiconghui commented Jun 3, 2024 •
edited
Loading

let
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@caiconghui
Copy link
Copy Markdown
Contributor Author

caiconghui commented Jun 3, 2024

run buildall

@caiconghui caiconghui mentioned this pull request Jun 3, 2024
Closed
@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 3, 2024

TPC-H: Total hot run time: 40197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a39460f874b4f14507cdc91de7205d939b16ef8, data reload: false

------ Round 1 ----------------------------------
q1	17662	4380	4283	4283
q2	2024	194	191	191
q3	10994	1254	1192	1192
q4	10704	793	704	704
q5	7647	2765	2699	2699
q6	220	138	144	138
q7	975	631	609	609
q8	9478	2073	2117	2073
q9	9211	6577	6631	6577
q10	9474	3758	3730	3730
q11	449	260	251	251
q12	548	236	225	225
q13	17777	2962	2966	2962
q14	255	223	217	217
q15	520	467	477	467
q16	520	400	384	384
q17	977	623	672	623
q18	7979	7531	7363	7363
q19	5786	1569	1499	1499
q20	646	311	320	311
q21	5016	3356	4080	3356
q22	432	343	352	343
Total cold run time: 119294 ms
Total hot run time: 40197 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4247	4263	4247
q2	376	261	267	261
q3	3034	2732	2787	2732
q4	1878	1648	1621	1621
q5	5256	5295	5281	5281
q6	224	130	131	130
q7	2188	1751	1764	1751
q8	3216	3317	3326	3317
q9	8303	8289	8346	8289
q10	3866	3583	3665	3583
q11	588	494	483	483
q12	753	612	624	612
q13	17447	3006	2986	2986
q14	302	266	273	266
q15	521	481	481	481
q16	486	428	444	428
q17	1793	1510	1483	1483
q18	7540	7536	7337	7337
q19	1736	1520	1660	1520
q20	1965	1778	1812	1778
q21	4899	4734	4824	4734
q22	613	551	544	544
Total cold run time: 71397 ms
Total hot run time: 53864 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 3, 2024

TPC-DS: Total hot run time: 173359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a39460f874b4f14507cdc91de7205d939b16ef8, data reload: false

query1	920	376	374	374
query2	6449	2386	2335	2335
query3	6658	207	205	205
query4	21842	17347	17405	17347
query5	4139	470	453	453
query6	259	159	174	159
query7	4584	312	300	300
query8	318	298	295	295
query9	8602	2398	2372	2372
query10	444	295	274	274
query11	10599	9868	9955	9868
query12	140	91	87	87
query13	1639	370	367	367
query14	9565	7741	7437	7437
query15	237	189	186	186
query16	7821	273	272	272
query17	1554	550	536	536
query18	1959	280	281	280
query19	199	163	163	163
query20	97	87	86	86
query21	211	144	127	127
query22	4245	4032	4009	4009
query23	33772	33014	33178	33014
query24	12314	2832	2773	2773
query25	709	381	380	380
query26	1802	161	161	161
query27	3009	330	336	330
query28	7554	2092	2066	2066
query29	1193	644	613	613
query30	274	151	156	151
query31	970	738	750	738
query32	97	53	56	53
query33	822	295	284	284
query34	1012	486	487	486
query35	738	620	601	601
query36	1086	924	975	924
query37	291	66	69	66
query38	2878	2766	2726	2726
query39	864	815	773	773
query40	289	126	125	125
query41	55	53	51	51
query42	120	94	129	94
query43	603	546	572	546
query44	1254	744	759	744
query45	201	166	165	165
query46	1073	723	711	711
query47	1862	1738	1782	1738
query48	383	309	309	309
query49	1188	402	407	402
query50	776	410	402	402
query51	6834	6746	6782	6746
query52	105	94	86	86
query53	359	299	288	288
query54	999	485	463	463
query55	74	72	72	72
query56	273	252	262	252
query57	1143	1041	1024	1024
query58	294	251	243	243
query59	3465	3272	3117	3117
query60	299	308	290	290
query61	94	87	91	87
query62	666	448	454	448
query63	317	295	301	295
query64	9785	2218	1742	1742
query65	3180	3129	3136	3129
query66	1377	328	327	327
query67	15289	14942	15134	14942
query68	4505	546	543	543
query69	592	450	367	367
query70	1095	1156	1026	1026
query71	427	282	286	282
query72	7166	5524	5587	5524
query73	771	337	325	325
query74	5929	5498	5480	5480
query75	3505	2678	2659	2659
query76	3009	902	910	902
query77	643	296	291	291
query78	10700	9966	9761	9761
query79	2785	510	510	510
query80	2275	466	471	466
query81	557	235	220	220
query82	1123	108	101	101
query83	308	169	175	169
query84	272	92	90	90
query85	1532	336	258	258
query86	482	331	321	321
query87	3261	3089	3059	3059
query88	3806	2457	2464	2457
query89	476	394	370	370
query90	1798	185	191	185
query91	126	96	98	96
query92	68	47	47	47
query93	2884	519	503	503
query94	1224	196	189	189
query95	405	316	316	316
query96	597	274	268	268
query97	3259	3029	3025	3025
query98	246	221	218	218
query99	1290	842	822	822
Total cold run time: 281339 ms
Total hot run time: 173359 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 3, 2024

ClickBench: Total hot run time: 30.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a39460f874b4f14507cdc91de7205d939b16ef8, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.05	0.04
query4	1.68	0.08	0.09
query5	0.50	0.47	0.47
query6	1.14	0.72	0.73
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.13	0.13
query13	0.59	0.59	0.59
query14	0.77	0.78	0.79
query15	0.84	0.82	0.81
query16	0.35	0.37	0.36
query17	0.99	0.98	0.98
query18	0.20	0.27	0.22
query19	1.88	1.67	1.68
query20	0.01	0.01	0.01
query21	15.73	0.67	0.66
query22	4.42	6.76	1.98
query23	18.29	1.40	1.23
query24	1.62	0.33	0.21
query25	0.16	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.09	0.07
query28	13.31	1.01	0.99
query29	13.20	3.33	3.30
query30	0.24	0.06	0.06
query31	2.87	0.39	0.39
query32	3.28	0.48	0.48
query33	2.92	2.91	2.90
query34	17.22	4.37	4.43
query35	4.53	4.53	4.56
query36	0.66	0.46	0.46
query37	0.17	0.16	0.16
query38	0.17	0.15	0.16
query39	0.05	0.03	0.03
query40	0.16	0.15	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.31 s
Total hot run time: 30.62 s

@caiconghui
Copy link
Copy Markdown
Contributor Author

caiconghui commented Jun 4, 2024

run external

1 similar comment
@caiconghui
Copy link
Copy Markdown
Contributor Author

caiconghui commented Jun 4, 2024

run external

GoGoWen
GoGoWen approved these changes Jun 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@GoGoWen GoGoWen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 4, 2024

PR approved by anyone and no changes requested.

@caiconghui caiconghui merged commit f2d8dd4 into master Jun 4, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@caiconghui @seawinde
[fix](stream_load) fix stream load may failed caused by column name w…
6289ec4 
…ith keyword (apache#35822)

let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved 

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
@morningman morningman mentioned this pull request Jun 6, 2024
Merged
morningman pushed a commit that referenced this pull request Jun 6, 2024
@caiconghui
[fix](keyword) let some keyword be non-reserved between old parser an…
fbfcd10 
…d new parser (#35938)

Followup #35822

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
dataroaring pushed a commit that referenced this pull request Jun 7, 2024
@caiconghui @dataroaring
[fix](stream_load) fix stream load may failed caused by column name w…
292ee7a 
…ith keyword (#35822)

let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved 

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
dataroaring pushed a commit that referenced this pull request Jun 7, 2024
@caiconghui @dataroaring
[fix](keyword) let some keyword be non-reserved between old parser an…
3fc552f 
…d new parser (#35938)

Followup #35822

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
caiconghui added a commit that referenced this pull request Jul 16, 2024
@caiconghui
[fix](stream_load) fix stream load may failed caused by column name w…
a1899eb 
…ith keyword (#35822)

let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved 

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
caiconghui added a commit that referenced this pull request Jul 16, 2024
@caiconghui
[fix](keyword) let some keyword be non-reserved between old parser an…
ae62284 
…d new parser (#35938)

Followup #35822

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
@caiconghui caiconghui mentioned this pull request Jul 16, 2024
Merged
caiconghui added a commit that referenced this pull request Jul 16, 2024
@caiconghui
[fix](keyword) let some keyword be non-reserved between old parser an…
f87b9f1 
…d new parser (#35938)

Followup #35822

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@caiconghui
[fix](stream_load) fix stream load may failed caused by column name w…
9ff129b 
…ith keyword (#35822) (#37890)

#35938 #35822 
let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@caiconghui
[fix](stream_load) fix stream load may failed caused by column name w…
735e072 
…ith keyword (apache#35822)

let
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
@dataroaring dataroaring deleted the stream_load_error branch January 19, 2025 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@weizuo93 weizuo93 weizuo93 approved these changes

@GoGoWen GoGoWen GoGoWen approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@caiconghui @doris-robot @weizuo93 @GoGoWen @morningman @yiguolei @dataroaring