Skip to content

branch-3.0: [fix](nereids) project distinct to agg checker exclude window function #54133#54541

Merged
dataroaring merged 2 commits intoapache:branch-3.0from
yujun777:pick_54133_to_upstream_branch-3.0
Aug 12, 2025
Merged

branch-3.0: [fix](nereids) project distinct to agg checker exclude window function #54133#54541
dataroaring merged 2 commits intoapache:branch-3.0from
yujun777:pick_54133_to_upstream_branch-3.0

Conversation

@yujun777
Copy link
Copy Markdown
Contributor

cherry pick from #54133

@yujun777 yujun777 requested a review from dataroaring as a code owner August 11, 2025 02:36
@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 39752 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b416840add880d782ebf446c2c033e68da83eadf, data reload: false

------ Round 1 ----------------------------------
q1	17743	7067	6806	6806
q2	2088	167	153	153
q3	10635	1132	1076	1076
q4	10245	708	664	664
q5	7784	2847	2779	2779
q6	220	141	136	136
q7	972	634	603	603
q8	9346	1916	2020	1916
q9	6672	6377	6415	6377
q10	7064	2238	2278	2238
q11	470	262	252	252
q12	401	212	227	212
q13	17812	2985	2983	2983
q14	235	210	209	209
q15	518	459	462	459
q16	476	385	372	372
q17	973	612	548	548
q18	7179	6675	6591	6591
q19	1386	1085	1039	1039
q20	476	200	207	200
q21	4198	3145	3170	3145
q22	1112	994	1009	994
Total cold run time: 108005 ms
Total hot run time: 39752 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6904	6974	6822	6822
q2	323	228	231	228
q3	2973	2979	2961	2961
q4	2053	1873	1857	1857
q5	5714	5779	5713	5713
q6	209	129	134	129
q7	2234	1857	1848	1848
q8	3341	3643	3456	3456
q9	8826	8898	8853	8853
q10	3594	3528	3539	3528
q11	599	506	498	498
q12	806	596	598	596
q13	8404	3164	3231	3164
q14	318	283	281	281
q15	527	473	469	469
q16	510	450	446	446
q17	1827	1636	1633	1633
q18	8282	7773	7789	7773
q19	1664	1506	1499	1499
q20	2071	1860	1882	1860
q21	5362	5043	5202	5043
q22	1142	1093	1029	1029
Total cold run time: 67683 ms
Total hot run time: 59686 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 197518 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b416840add880d782ebf446c2c033e68da83eadf, data reload: false

query1	1277	902	892	892
query2	6271	1948	1898	1898
query3	10952	4531	4690	4531
query4	33219	23528	23505	23505
query5	3628	454	435	435
query6	265	175	180	175
query7	3994	312	319	312
query8	285	233	223	223
query9	9627	2580	2553	2553
query10	479	271	263	263
query11	18019	15335	15218	15218
query12	153	99	106	99
query13	1556	455	421	421
query14	8528	7360	7725	7360
query15	285	188	189	188
query16	8129	514	527	514
query17	1728	588	579	579
query18	2191	342	328	328
query19	393	169	163	163
query20	128	119	113	113
query21	209	110	108	108
query22	4668	4432	4381	4381
query23	35063	34428	34202	34202
query24	11601	2985	2992	2985
query25	700	423	433	423
query26	1676	179	189	179
query27	2830	355	363	355
query28	7498	2166	2153	2153
query29	1000	477	470	470
query30	277	159	164	159
query31	1068	850	859	850
query32	109	57	58	57
query33	760	289	292	289
query34	926	513	502	502
query35	920	740	727	727
query36	1111	932	963	932
query37	141	69	63	63
query38	4135	3967	4019	3967
query39	1535	1522	1459	1459
query40	256	100	98	98
query41	51	47	49	47
query42	119	103	102	102
query43	505	490	495	490
query44	1272	797	807	797
query45	190	182	173	173
query46	1190	745	733	733
query47	1992	1919	1893	1893
query48	469	377	378	377
query49	1023	413	413	413
query50	860	448	436	436
query51	7291	7432	7311	7311
query52	100	88	94	88
query53	266	183	185	183
query54	1194	457	493	457
query55	80	76	77	76
query56	268	256	242	242
query57	1306	1214	1202	1202
query58	243	209	220	209
query59	3221	2968	2934	2934
query60	292	255	260	255
query61	116	112	104	104
query62	879	703	691	691
query63	231	196	190	190
query64	4942	676	644	644
query65	3339	3278	3305	3278
query66	1118	294	302	294
query67	16016	15413	15546	15413
query68	4879	571	577	571
query69	429	289	260	260
query70	1188	1123	1103	1103
query71	357	256	256	256
query72	6201	3794	3959	3794
query73	745	345	342	342
query74	10802	9097	9157	9097
query75	3398	2628	2685	2628
query76	2662	1110	1113	1110
query77	416	288	274	274
query78	10619	9546	9527	9527
query79	1820	582	591	582
query80	1172	417	420	417
query81	569	223	223	223
query82	939	93	87	87
query83	226	142	139	139
query84	239	73	84	73
query85	1389	306	293	293
query86	452	306	309	306
query87	4442	4236	4282	4236
query88	3682	2363	2339	2339
query89	421	295	295	295
query90	1991	183	183	183
query91	177	147	151	147
query92	63	50	51	50
query93	2408	547	552	547
query94	947	302	299	299
query95	364	255	254	254
query96	612	280	279	279
query97	3250	3146	3190	3146
query98	231	204	192	192
query99	1524	1308	1327	1308
Total cold run time: 304777 ms
Total hot run time: 197518 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 29.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b416840add880d782ebf446c2c033e68da83eadf, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.52	0.49	0.51
query6	1.12	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.56	0.51	0.50
query10	0.55	0.57	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	0.77	0.81	0.80
query15	0.86	0.84	0.82
query16	0.40	0.38	0.40
query17	1.06	1.02	1.05
query18	0.24	0.21	0.22
query19	1.96	1.81	1.88
query20	0.02	0.01	0.02
query21	15.41	0.59	0.59
query22	2.30	1.74	1.84
query23	17.02	0.98	0.74
query24	3.35	0.64	1.34
query25	0.35	0.07	0.07
query26	0.35	0.13	0.15
query27	0.05	0.04	0.06
query28	10.23	0.50	0.48
query29	12.60	3.29	3.31
query30	0.24	0.06	0.07
query31	2.85	0.38	0.38
query32	3.25	0.46	0.46
query33	2.97	3.02	3.00
query34	17.21	4.53	4.53
query35	4.59	4.57	4.62
query36	0.67	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.89 s
Total hot run time: 29.92 s

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 76.47% (13/17) 🎉
Increment coverage report
Complete coverage report

@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 39685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2754d50cdf37d1599dcd8f3635c47c0dcc329d5, data reload: false

------ Round 1 ----------------------------------
q1	17592	6782	6630	6630
q2	2067	185	173	173
q3	12091	1140	1200	1140
q4	10218	724	719	719
q5	7737	2892	2776	2776
q6	214	137	132	132
q7	958	616	602	602
q8	9360	1968	1991	1968
q9	6563	6405	6442	6405
q10	6996	2244	2259	2244
q11	453	262	257	257
q12	400	204	209	204
q13	17766	2993	2984	2984
q14	234	200	228	200
q15	495	464	469	464
q16	484	384	374	374
q17	979	632	559	559
q18	7302	6736	6517	6517
q19	1406	1114	1069	1069
q20	475	203	202	202
q21	4042	3099	3180	3099
q22	1106	967	1001	967
Total cold run time: 108938 ms
Total hot run time: 39685 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6624	6544	6561	6544
q2	323	234	230	230
q3	2990	3037	2984	2984
q4	2039	1799	1794	1794
q5	5776	5700	5753	5700
q6	201	125	128	125
q7	2204	1784	1807	1784
q8	3340	3502	3507	3502
q9	8861	8895	8929	8895
q10	3599	3518	3513	3513
q11	586	488	513	488
q12	830	648	616	616
q13	3921	3179	3180	3179
q14	311	280	276	276
q15	520	464	467	464
q16	489	427	424	424
q17	1823	1652	1601	1601
q18	8180	7746	7706	7706
q19	4626	1664	1594	1594
q20	2083	1858	1896	1858
q21	5055	4832	4838	4832
q22	1112	1036	1004	1004
Total cold run time: 65493 ms
Total hot run time: 59113 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 189945 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2754d50cdf37d1599dcd8f3635c47c0dcc329d5, data reload: false

query1	969	376	400	376
query2	6515	1871	1889	1871
query3	6715	215	217	215
query4	34157	23427	23855	23427
query5	4558	462	438	438
query6	274	173	177	173
query7	4631	302	309	302
query8	279	227	222	222
query9	9577	2604	2569	2569
query10	474	264	253	253
query11	18294	15225	15495	15225
query12	160	101	99	99
query13	1651	422	410	410
query14	9410	6688	7215	6688
query15	260	182	185	182
query16	8094	487	479	479
query17	1664	581	567	567
query18	2132	341	316	316
query19	367	161	162	161
query20	116	108	110	108
query21	204	105	107	105
query22	4403	4162	4163	4162
query23	34458	33444	33762	33444
query24	11776	2851	2821	2821
query25	724	420	413	413
query26	1890	173	167	167
query27	2991	347	348	347
query28	7915	2122	2134	2122
query29	1083	461	444	444
query30	322	160	158	158
query31	1035	804	800	800
query32	95	62	58	58
query33	809	308	306	306
query34	946	506	523	506
query35	877	740	708	708
query36	1095	933	926	926
query37	272	76	74	74
query38	3939	3816	3783	3783
query39	1507	1415	1437	1415
query40	289	103	105	103
query41	53	52	54	52
query42	111	101	103	101
query43	514	471	478	471
query44	1309	800	790	790
query45	183	168	171	168
query46	1152	711	708	708
query47	1924	1856	1816	1816
query48	497	395	380	380
query49	1337	416	410	410
query50	817	430	417	417
query51	7293	7136	7110	7110
query52	102	93	100	93
query53	258	187	189	187
query54	1107	468	467	467
query55	78	79	81	79
query56	270	268	258	258
query57	1328	1156	1152	1152
query58	243	211	243	211
query59	3035	2772	2744	2744
query60	304	248	250	248
query61	115	109	126	109
query62	879	663	691	663
query63	218	191	190	190
query64	5118	665	666	665
query65	3324	3186	3229	3186
query66	1421	311	309	309
query67	15941	15777	15493	15493
query68	4809	568	574	568
query69	433	273	265	265
query70	1152	1074	1083	1074
query71	331	260	255	255
query72	6383	3994	4134	3994
query73	740	342	357	342
query74	10268	9056	8993	8993
query75	3366	2625	2662	2625
query76	2891	1114	1047	1047
query77	380	299	271	271
query78	10466	9648	9640	9640
query79	2454	596	596	596
query80	993	436	432	432
query81	532	223	221	221
query82	686	91	90	90
query83	257	148	149	148
query84	244	75	78	75
query85	1640	304	290	290
query86	483	300	285	285
query87	4406	4284	4323	4284
query88	4414	2369	2330	2330
query89	400	296	300	296
query90	2046	194	186	186
query91	183	155	149	149
query92	63	52	50	50
query93	1667	559	550	550
query94	971	301	294	294
query95	359	260	259	259
query96	617	279	279	279
query97	3329	3139	3171	3139
query98	218	201	192	192
query99	1548	1321	1306	1306
Total cold run time: 304587 ms
Total hot run time: 189945 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 29.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2754d50cdf37d1599dcd8f3635c47c0dcc329d5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.06	0.07
query4	1.63	0.11	0.11
query5	0.52	0.51	0.50
query6	1.14	0.74	0.73
query7	0.02	0.03	0.01
query8	0.04	0.03	0.03
query9	0.55	0.49	0.50
query10	0.54	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.61
query14	0.76	0.82	0.82
query15	0.84	0.82	0.82
query16	0.38	0.38	0.39
query17	1.06	1.07	0.98
query18	0.22	0.22	0.22
query19	1.94	1.79	1.87
query20	0.02	0.01	0.01
query21	15.39	0.59	0.60
query22	2.44	1.95	1.40
query23	16.73	0.90	0.74
query24	3.48	1.29	0.87
query25	0.18	0.13	0.10
query26	0.48	0.14	0.14
query27	0.04	0.04	0.05
query28	10.00	0.52	0.47
query29	12.54	3.24	3.22
query30	0.25	0.06	0.06
query31	2.88	0.39	0.39
query32	3.24	0.47	0.45
query33	2.96	2.98	3.00
query34	17.15	4.55	4.53
query35	4.62	4.53	4.53
query36	0.69	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.01
query43	0.03	0.03	0.04
Total cold run time: 104.4 s
Total hot run time: 29.66 s

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 76.47% (13/17) 🎉
Increment coverage report
Complete coverage report

Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2025
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit d646d2c into apache:branch-3.0 Aug 12, 2025
24 of 25 checks passed
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants