Skip to content

branch-3.0: [enhancement](nereids) improve lots of values in insert into values statement #40202 #51925 #52802#53185

Closed
924060929 wants to merge 2 commits intoapache:branch-3.0from
924060929:branch-3.0-fast-insert-into
Closed

branch-3.0: [enhancement](nereids) improve lots of values in insert into values statement #40202 #51925 #52802#53185
924060929 wants to merge 2 commits intoapache:branch-3.0from
924060929:branch-3.0-fast-insert-into

Conversation

@924060929
Copy link
Copy Markdown
Contributor

@924060929 924060929 commented Jul 14, 2025

cherry pick from #40202 #51925 #52802

924060929 added 2 commits July 14, 2025 11:28
@924060929
branch-3.0: [enhancement](nereids) improve lots of values in insert i…
8be261b 
…nto values statement (apache#40202) (apache#51925)

cherry pick from apache#40202 and apache#51925
@924060929
[fix](nereids) fix insert into values throw 'Invalid call to sql on u…
c261af6 
…nbound object' and 'Insert has filtered data in strict mode' exception (apache#52802)

1. fix `Invalid call to sql on unbound object` when use `interval`,
introduced by apache#40202
```sql
CREATE TABLE `test_insert_cast_interval` (
  `id` int NULL,
  `dt` date NULL
) ENGINE=OLAP
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
INSERT INTO test_insert_cast_interval values(1, date_floor('2020-02-02', interval 1 second));

(1105, 'errCode = 2, detailMessage = Invalid call to sql on unbound object')
```

2. fix `Insert has filtered data in strict mode`, introduced by apache#49116
```sql
CREATE TABLE `test_insert_more_string` (
  `r_regionkey` int NULL,
  `r_name` varchar(25) NULL,
  `r_comment` varchar(152) NULL
) ENGINE=OLAP
DISTRIBUTED BY HASH(`r_regionkey`) BUCKETS 1
PROPERTIES ( "replication_allocation" = "tag.location.default: 1");

insert into test_insert_more_string values (3, "akljalkjbalkjsldkrjewokjfalksdjflaksjfdlaskjfalsdkfjalsdfjkasfdl", "aa");

(1105, 'errCode = 2, detailMessage = Insert has filtered data in strict mode')
```

(cherry picked from commit 2c01f69)
@924060929 924060929 requested a review from dataroaring as a code owner July 14, 2025 04:57
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Jul 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Copy Markdown
Contributor Author

924060929 commented Jul 14, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 14, 2025

TPC-H: Total hot run time: 39841 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c261af648cc96c438769758cc4ae21924c405e69, data reload: false

------ Round 1 ----------------------------------
q1	17757	7489	6644	6644
q2	2069	163	153	153
q3	10658	1130	1149	1130
q4	10503	776	712	712
q5	7712	2912	2846	2846
q6	215	132	133	132
q7	981	613	603	603
q8	9350	1923	1984	1923
q9	6628	6378	6391	6378
q10	7022	2286	2258	2258
q11	446	257	258	257
q12	392	215	213	213
q13	17795	2960	3016	2960
q14	227	201	206	201
q15	521	474	472	472
q16	520	375	386	375
q17	977	617	538	538
q18	7397	6637	6669	6637
q19	1389	1066	1069	1066
q20	464	194	201	194
q21	3867	3128	3279	3128
q22	1111	1021	1064	1021
Total cold run time: 108001 ms
Total hot run time: 39841 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6653	6635	6617	6617
q2	345	240	231	231
q3	2989	2971	2935	2935
q4	2046	1861	1847	1847
q5	5744	5799	5781	5781
q6	215	130	130	130
q7	2268	1770	1764	1764
q8	3363	3552	3486	3486
q9	8818	8959	9027	8959
q10	3592	3604	3582	3582
q11	586	494	518	494
q12	859	608	642	608
q13	8597	3120	3176	3120
q14	299	264	286	264
q15	536	470	478	470
q16	497	438	452	438
q17	1850	1733	1622	1622
q18	8271	7707	7672	7672
q19	1665	1473	1532	1473
q20	2120	1861	1871	1861
q21	5061	4992	5075	4992
q22	1152	1057	1037	1037
Total cold run time: 67526 ms
Total hot run time: 59383 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 14, 2025

TPC-DS: Total hot run time: 196798 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c261af648cc96c438769758cc4ae21924c405e69, data reload: false

query1	1278	888	899	888
query2	6256	2000	1901	1901
query3	10985	4618	4627	4618
query4	33249	23490	23928	23490
query5	4460	461	437	437
query6	277	189	190	189
query7	4009	321	327	321
query8	285	223	225	223
query9	9491	2599	2577	2577
query10	476	259	246	246
query11	18119	15271	15181	15181
query12	145	98	97	97
query13	1531	422	403	403
query14	8616	6782	7617	6782
query15	243	193	188	188
query16	8032	499	508	499
query17	1599	593	615	593
query18	2174	327	335	327
query19	215	173	162	162
query20	123	122	115	115
query21	216	106	104	104
query22	4660	4471	4430	4430
query23	35278	34477	34011	34011
query24	11178	2903	2921	2903
query25	673	421	425	421
query26	1522	175	171	171
query27	2789	343	339	339
query28	7878	2183	2172	2172
query29	952	484	445	445
query30	262	169	166	166
query31	1054	825	804	804
query32	100	58	57	57
query33	784	306	320	306
query34	930	520	514	514
query35	878	737	738	737
query36	1105	951	950	950
query37	138	77	65	65
query38	4138	4008	4061	4008
query39	1696	1488	1489	1488
query40	270	98	99	98
query41	47	53	50	50
query42	115	99	102	99
query43	538	495	488	488
query44	1265	816	806	806
query45	195	166	170	166
query46	1139	723	716	716
query47	2011	1944	1940	1940
query48	469	367	380	367
query49	977	408	398	398
query50	839	428	416	416
query51	7665	7400	7292	7292
query52	103	87	92	87
query53	263	183	181	181
query54	1347	478	492	478
query55	76	78	73	73
query56	268	268	248	248
query57	1340	1191	1200	1191
query58	228	238	243	238
query59	3298	3234	3024	3024
query60	280	264	236	236
query61	113	104	143	104
query62	867	677	696	677
query63	213	189	187	187
query64	5398	678	691	678
query65	3380	3283	3303	3283
query66	1346	298	301	298
query67	16191	15395	15489	15395
query68	4500	562	577	562
query69	423	259	263	259
query70	1172	1125	1095	1095
query71	348	248	253	248
query72	6194	4005	4157	4005
query73	779	356	344	344
query74	10325	9095	8995	8995
query75	3395	2614	2657	2614
query76	2697	983	980	980
query77	372	264	271	264
query78	10549	9589	9551	9551
query79	2012	615	593	593
query80	1083	426	411	411
query81	543	218	220	218
query82	935	87	90	87
query83	250	145	142	142
query84	234	82	81	81
query85	1376	296	297	296
query86	445	305	286	286
query87	4436	4222	4251	4222
query88	3932	2365	2365	2365
query89	417	291	303	291
query90	1981	185	183	183
query91	182	147	146	146
query92	69	49	50	49
query93	2463	558	547	547
query94	877	287	279	279
query95	364	261	243	243
query96	627	283	281	281
query97	3277	3147	3155	3147
query98	224	204	195	195
query99	1486	1305	1277	1277
Total cold run time: 306002 ms
Total hot run time: 196798 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 14, 2025

ClickBench: Total hot run time: 29.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c261af648cc96c438769758cc4ae21924c405e69, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.53	0.51	0.50
query6	1.17	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.05
query9	0.56	0.51	0.50
query10	0.55	0.57	0.57
query11	0.14	0.11	0.11
query12	0.14	0.13	0.12
query13	0.61	0.59	0.61
query14	0.77	0.80	0.80
query15	0.86	0.81	0.82
query16	0.38	0.37	0.39
query17	1.06	1.06	1.07
query18	0.24	0.21	0.22
query19	1.95	1.86	1.79
query20	0.01	0.01	0.01
query21	15.40	0.57	0.56
query22	2.83	2.66	1.05
query23	17.03	0.87	0.82
query24	2.94	1.28	1.57
query25	0.11	0.09	0.28
query26	0.51	0.14	0.13
query27	0.04	0.03	0.03
query28	9.73	0.50	0.46
query29	12.56	3.22	3.26
query30	0.24	0.06	0.05
query31	2.86	0.39	0.37
query32	3.25	0.46	0.46
query33	2.97	3.03	2.98
query34	17.18	4.55	4.49
query35	4.54	4.62	4.55
query36	0.66	0.48	0.47
query37	0.08	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.3 s
Total hot run time: 29.73 s

@924060929 924060929 marked this pull request as draft July 14, 2025 08:40
@924060929 924060929 closed this Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

@lide-reed lide-reed Awaiting requested review from lide-reed lide-reed will be requested when the pull request is marked ready for review lide-reed is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@924060929 @Thearas @doris-robot