[opt](nereids) refine left semi/anti shortcut cost by xzj7019 · Pull Request #37060 · apache/doris

xzj7019 · 2024-07-01T04:31:49Z

Proposed changes

Current left semi/anti join's cost doesn't consider be's short-cut opt, this pr will refine this part of computing and distinguish the left and right join costs.

doris-robot · 2024-07-01T04:31:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xzj7019 · 2024-07-01T04:32:25Z

run buildall

doris-robot · 2024-07-01T05:00:26Z

TPC-H: Total hot run time: 40092 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit daf1f791e5a5c727002efd5df2a8b43f5be11a75, data reload: false

------ Round 1 ----------------------------------
q1	17609	4537	4349	4349
q2	2026	197	206	197
q3	10449	1199	1081	1081
q4	10179	743	888	743
q5	7509	2674	2702	2674
q6	223	140	137	137
q7	957	601	607	601
q8	9240	2104	2094	2094
q9	8944	6555	6525	6525
q10	9038	3728	3754	3728
q11	451	234	234	234
q12	485	233	230	230
q13	17775	3011	3015	3011
q14	259	234	220	220
q15	535	496	483	483
q16	550	374	369	369
q17	979	610	688	610
q18	8102	7498	7514	7498
q19	4922	1575	1338	1338
q20	665	322	317	317
q21	5048	3321	3929	3321
q22	396	334	332	332
Total cold run time: 116341 ms
Total hot run time: 40092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4210	4248	4210
q2	368	269	265	265
q3	2990	2905	2851	2851
q4	2033	1716	1726	1716
q5	5647	5496	5479	5479
q6	229	137	132	132
q7	2268	1855	1855	1855
q8	3314	3425	3454	3425
q9	8731	8748	8817	8748
q10	4206	3879	3776	3776
q11	599	508	551	508
q12	839	657	652	652
q13	17207	3126	3235	3126
q14	307	289	290	289
q15	541	482	506	482
q16	492	431	434	431
q17	1826	1542	1502	1502
q18	8225	8043	7968	7968
q19	1846	1589	1530	1530
q20	2097	1914	1829	1829
q21	9537	4968	4763	4763
q22	676	560	576	560
Total cold run time: 78383 ms
Total hot run time: 56097 ms

doris-robot · 2024-07-01T05:11:32Z

TPC-DS: Total hot run time: 174066 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit daf1f791e5a5c727002efd5df2a8b43f5be11a75, data reload: false

query1	928	390	387	387
query2	6456	2292	2308	2292
query3	6633	206	219	206
query4	19544	17638	17344	17344
query5	3667	482	479	479
query6	250	168	160	160
query7	4585	292	302	292
query8	300	275	271	271
query9	8674	2392	2366	2366
query10	556	308	276	276
query11	10739	10108	10084	10084
query12	112	96	83	83
query13	1651	368	360	360
query14	10207	7617	7527	7527
query15	224	183	179	179
query16	7747	326	314	314
query17	1797	526	519	519
query18	1961	268	262	262
query19	194	144	148	144
query20	90	79	80	79
query21	213	132	132	132
query22	4463	4334	4394	4334
query23	33755	33684	33812	33684
query24	11930	2894	2866	2866
query25	664	375	397	375
query26	1678	157	155	155
query27	2930	322	330	322
query28	7486	2141	2116	2116
query29	1037	639	629	629
query30	252	157	156	156
query31	1028	759	753	753
query32	92	58	53	53
query33	757	291	272	272
query34	1030	478	503	478
query35	799	641	634	634
query36	1158	986	985	985
query37	179	79	83	79
query38	2986	2873	2832	2832
query39	905	835	859	835
query40	261	130	124	124
query41	55	54	58	54
query42	118	105	105	105
query43	611	538	541	538
query44	1226	739	725	725
query45	197	162	165	162
query46	1080	739	720	720
query47	1913	1795	1814	1795
query48	361	305	295	295
query49	921	442	411	411
query50	776	384	379	379
query51	7067	6879	6832	6832
query52	98	101	89	89
query53	359	289	286	286
query54	931	434	448	434
query55	74	73	73	73
query56	283	267	257	257
query57	1142	1040	1039	1039
query58	268	250	246	246
query59	3441	3162	3093	3093
query60	306	299	277	277
query61	91	90	91	90
query62	605	464	439	439
query63	314	283	289	283
query64	9671	2226	1731	1731
query65	3150	3125	3109	3109
query66	1043	333	344	333
query67	15557	15094	15132	15094
query68	4621	535	531	531
query69	466	305	304	304
query70	1166	1155	1138	1138
query71	410	272	273	272
query72	7178	5243	5379	5243
query73	735	321	324	321
query74	6063	5550	5478	5478
query75	3421	2677	2701	2677
query76	2746	875	918	875
query77	422	300	290	290
query78	9655	9048	8878	8878
query79	2583	520	513	513
query80	2137	463	464	463
query81	578	219	221	219
query82	920	110	102	102
query83	286	172	169	169
query84	267	84	89	84
query85	2109	290	288	288
query86	458	317	320	317
query87	3305	3156	3137	3137
query88	4189	2368	2350	2350
query89	473	374	392	374
query90	1788	189	188	188
query91	127	100	103	100
query92	58	49	48	48
query93	3569	511	493	493
query94	1093	210	209	209
query95	399	304	316	304
query96	610	265	266	265
query97	3267	3091	3082	3082
query98	225	193	202	193
query99	1302	834	839	834
Total cold run time: 278202 ms
Total hot run time: 174066 ms

doris-robot · 2024-07-01T05:17:03Z

ClickBench: Total hot run time: 30.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit daf1f791e5a5c727002efd5df2a8b43f5be11a75, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.07	0.07
query5	0.58	0.48	0.48
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.16	0.13	0.12
query13	0.59	0.60	0.60
query14	0.75	0.79	0.77
query15	0.83	0.82	0.81
query16	0.37	0.36	0.36
query17	0.95	0.96	0.98
query18	0.22	0.25	0.25
query19	1.87	1.67	1.71
query20	0.01	0.01	0.00
query21	15.43	0.75	0.66
query22	4.24	7.45	1.70
query23	18.29	1.40	1.25
query24	2.11	0.23	0.21
query25	0.15	0.08	0.08
query26	0.28	0.17	0.18
query27	0.07	0.08	0.08
query28	13.27	1.02	0.99
query29	12.60	3.27	3.18
query30	0.25	0.06	0.05
query31	2.87	0.40	0.39
query32	3.27	0.47	0.49
query33	2.83	2.95	2.87
query34	17.34	4.43	4.44
query35	4.49	4.48	4.46
query36	0.64	0.47	0.46
query37	0.19	0.15	0.16
query38	0.14	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.03
Total cold run time: 109.67 s
Total hot run time: 30.15 s

xzj7019 · 2024-07-01T06:40:37Z

run buidall

xzj7019 · 2024-07-01T06:50:40Z

run buildall

doris-robot · 2024-07-01T07:19:50Z

TPC-H: Total hot run time: 40473 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 014567a24f4467d06d1bb31f6113b64fbc25b237, data reload: false

------ Round 1 ----------------------------------
q1	17607	4397	4223	4223
q2	2020	191	188	188
q3	10489	1248	1136	1136
q4	10212	702	773	702
q5	7475	2613	2636	2613
q6	228	142	138	138
q7	946	608	619	608
q8	9223	2065	2076	2065
q9	9069	6488	6477	6477
q10	9058	3768	3741	3741
q11	461	237	233	233
q12	518	248	235	235
q13	17773	3034	3007	3007
q14	260	238	238	238
q15	529	487	480	480
q16	490	381	381	381
q17	966	682	677	677
q18	7986	7482	7356	7356
q19	7072	1497	1447	1447
q20	686	332	338	332
q21	4985	3848	3908	3848
q22	421	348	352	348
Total cold run time: 118474 ms
Total hot run time: 40473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4347	4241	4220	4220
q2	383	272	269	269
q3	3090	2894	2963	2894
q4	1971	1738	1801	1738
q5	5478	5441	5455	5441
q6	222	137	131	131
q7	2241	1866	1834	1834
q8	3317	3426	3419	3419
q9	8734	8651	8754	8651
q10	4379	3733	3802	3733
q11	613	494	505	494
q12	826	650	659	650
q13	16961	3151	3191	3151
q14	321	282	280	280
q15	524	517	512	512
q16	498	443	432	432
q17	1820	1506	1503	1503
q18	8235	8012	7792	7792
q19	1807	1671	1589	1589
q20	2151	1889	1851	1851
q21	9506	4936	4734	4734
q22	633	584	548	548
Total cold run time: 78057 ms
Total hot run time: 55866 ms

doris-robot · 2024-07-01T07:31:09Z

TPC-DS: Total hot run time: 173772 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 014567a24f4467d06d1bb31f6113b64fbc25b237, data reload: false

query1	934	384	372	372
query2	6440	2336	2426	2336
query3	6632	204	211	204
query4	19481	17419	17419	17419
query5	3661	507	468	468
query6	282	167	161	161
query7	4589	296	308	296
query8	303	300	301	300
query9	8741	2386	2369	2369
query10	571	325	290	290
query11	10406	10032	9940	9940
query12	116	86	91	86
query13	1641	376	371	371
query14	10150	7728	7461	7461
query15	232	186	187	186
query16	7759	323	322	322
query17	1787	553	542	542
query18	1966	281	278	278
query19	199	170	163	163
query20	89	87	85	85
query21	213	136	131	131
query22	4365	4094	4155	4094
query23	34050	33833	33730	33730
query24	10918	2892	2876	2876
query25	626	403	395	395
query26	723	166	162	162
query27	2259	327	322	322
query28	6050	2132	2121	2121
query29	929	672	652	652
query30	255	158	152	152
query31	963	783	788	783
query32	94	56	56	56
query33	729	325	310	310
query34	880	484	493	484
query35	771	647	661	647
query36	1163	982	977	977
query37	147	83	85	83
query38	2894	2870	2858	2858
query39	911	856	846	846
query40	210	144	134	134
query41	60	56	56	56
query42	111	103	109	103
query43	608	543	542	542
query44	1138	738	811	738
query45	194	173	158	158
query46	1074	682	740	682
query47	1838	1736	1756	1736
query48	364	309	304	304
query49	829	408	406	406
query50	775	377	390	377
query51	6773	6810	6704	6704
query52	108	96	102	96
query53	363	294	292	292
query54	914	441	431	431
query55	72	76	74	74
query56	282	262	271	262
query57	1133	1079	1030	1030
query58	264	274	243	243
query59	3471	3129	3493	3129
query60	307	280	282	280
query61	92	93	94	93
query62	600	447	452	447
query63	324	291	291	291
query64	8834	2262	1780	1780
query65	3290	3130	3134	3130
query66	813	331	336	331
query67	15687	14849	14809	14809
query68	8671	539	540	539
query69	741	418	327	327
query70	1434	1143	1153	1143
query71	505	280	280	280
query72	9060	5667	5079	5079
query73	2238	334	334	334
query74	5836	5541	5504	5504
query75	5319	2716	2657	2657
query76	5154	1011	922	922
query77	787	311	309	309
query78	9585	8990	9253	8990
query79	8028	519	517	517
query80	2054	475	478	475
query81	566	218	222	218
query82	272	105	104	104
query83	296	173	168	168
query84	276	83	83	83
query85	890	332	264	264
query86	359	320	330	320
query87	3282	3103	3089	3089
query88	4897	2374	2384	2374
query89	545	387	378	378
query90	2038	189	193	189
query91	127	98	98	98
query92	58	50	51	50
query93	5835	512	498	498
query94	1370	214	209	209
query95	411	320	314	314
query96	605	271	262	262
query97	3228	3050	3070	3050
query98	223	195	193	193
query99	1074	859	849	849
Total cold run time: 291211 ms
Total hot run time: 173772 ms

doris-robot · 2024-07-01T07:36:41Z

ClickBench: Total hot run time: 30.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 014567a24f4467d06d1bb31f6113b64fbc25b237, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.48	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.48	0.50
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.79	0.77
query15	0.83	0.81	0.81
query16	0.36	0.35	0.37
query17	1.05	1.04	1.04
query18	0.22	0.27	0.24
query19	1.82	1.76	1.74
query20	0.02	0.00	0.00
query21	15.45	0.76	0.67
query22	4.56	7.07	1.79
query23	18.30	1.41	1.26
query24	2.29	0.23	0.24
query25	0.16	0.09	0.08
query26	0.26	0.18	0.18
query27	0.08	0.08	0.08
query28	13.30	1.01	0.99
query29	12.64	3.30	3.30
query30	0.26	0.06	0.06
query31	2.87	0.38	0.39
query32	3.29	0.48	0.47
query33	2.92	2.90	2.90
query34	17.08	4.41	4.41
query35	4.46	4.52	4.50
query36	0.66	0.46	0.47
query37	0.18	0.15	0.16
query38	0.15	0.16	0.14
query39	0.04	0.04	0.04
query40	0.16	0.15	0.15
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.05
Total cold run time: 110.07 s
Total hot run time: 30.59 s

xzj7019 · 2024-07-01T07:45:28Z

run buildall

doris-robot · 2024-07-01T08:48:13Z

TPC-H: Total hot run time: 39918 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19e151152c45885dc3dae2acaed41a6febfb14a9, data reload: false

------ Round 1 ----------------------------------
q1	17672	4650	4289	4289
q2	2021	202	198	198
q3	10497	1288	1125	1125
q4	10214	748	803	748
q5	7503	2662	2675	2662
q6	218	138	138	138
q7	960	607	596	596
q8	9235	2082	2076	2076
q9	8965	6496	6468	6468
q10	8950	3756	3735	3735
q11	460	242	239	239
q12	526	237	243	237
q13	17759	3021	2977	2977
q14	276	224	231	224
q15	528	501	490	490
q16	494	379	373	373
q17	987	784	698	698
q18	8034	7384	7327	7327
q19	7095	1542	1469	1469
q20	683	331	321	321
q21	4945	3240	3198	3198
q22	394	330	350	330
Total cold run time: 118416 ms
Total hot run time: 39918 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4437	4380	4244	4244
q2	374	278	268	268
q3	2967	2902	2935	2902
q4	2015	1666	1788	1666
q5	5610	5505	5500	5500
q6	235	132	132	132
q7	2177	1885	1852	1852
q8	3228	3421	3429	3421
q9	8730	8630	8901	8630
q10	4113	3905	3752	3752
q11	613	500	502	500
q12	838	646	639	639
q13	16136	3164	3216	3164
q14	314	289	283	283
q15	528	507	486	486
q16	474	412	434	412
q17	1858	1546	1538	1538
q18	8211	7938	7798	7798
q19	1813	1626	1541	1541
q20	2145	1935	1853	1853
q21	5159	5004	4852	4852
q22	633	592	590	590
Total cold run time: 72608 ms
Total hot run time: 56023 ms

doris-robot · 2024-07-01T08:59:08Z

TPC-DS: Total hot run time: 172465 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19e151152c45885dc3dae2acaed41a6febfb14a9, data reload: false

query1	911	380	387	380
query2	6471	2353	2358	2353
query3	6648	214	218	214
query4	19122	17275	17191	17191
query5	3692	487	479	479
query6	264	192	172	172
query7	4608	298	311	298
query8	320	293	289	289
query9	8441	2396	2366	2366
query10	583	306	301	301
query11	10672	10111	10124	10111
query12	113	84	83	83
query13	1638	375	372	372
query14	10217	7605	7091	7091
query15	236	193	192	192
query16	7774	359	298	298
query17	1760	547	521	521
query18	1956	269	260	260
query19	198	144	147	144
query20	91	83	81	81
query21	206	139	126	126
query22	4413	3995	4002	3995
query23	33809	33829	33699	33699
query24	11202	2902	2832	2832
query25	592	385	384	384
query26	772	161	154	154
query27	2352	327	336	327
query28	6354	2124	2129	2124
query29	893	651	622	622
query30	244	156	152	152
query31	982	757	771	757
query32	92	53	58	53
query33	770	296	285	285
query34	1042	485	488	485
query35	780	625	634	625
query36	1151	965	984	965
query37	142	85	85	85
query38	2949	2834	2813	2813
query39	863	850	820	820
query40	209	133	130	130
query41	53	50	53	50
query42	110	105	105	105
query43	615	578	548	548
query44	1186	718	733	718
query45	193	167	159	159
query46	1071	711	742	711
query47	1873	1750	1756	1750
query48	374	303	305	303
query49	862	404	408	404
query50	771	399	384	384
query51	6812	6593	6786	6593
query52	97	93	97	93
query53	358	297	298	297
query54	894	461	442	442
query55	74	73	101	73
query56	285	265	270	265
query57	1120	1067	1044	1044
query58	249	247	281	247
query59	3474	3138	3206	3138
query60	310	281	293	281
query61	91	91	87	87
query62	620	443	437	437
query63	322	286	285	285
query64	8559	2276	1734	1734
query65	3211	3079	3084	3079
query66	746	320	323	320
query67	15444	15186	14729	14729
query68	4486	529	532	529
query69	480	308	306	306
query70	1133	1156	1108	1108
query71	411	289	304	289
query72	6949	5232	5184	5184
query73	744	326	324	324
query74	5883	5490	5493	5490
query75	3427	2632	2646	2632
query76	2742	908	983	908
query77	452	307	313	307
query78	9698	9029	8884	8884
query79	2761	508	524	508
query80	2199	471	531	471
query81	593	218	216	216
query82	999	113	111	111
query83	332	173	169	169
query84	269	94	92	92
query85	1429	284	272	272
query86	468	299	303	299
query87	3345	3068	3141	3068
query88	3912	2354	2357	2354
query89	472	372	383	372
query90	1801	191	183	183
query91	126	97	100	97
query92	58	50	49	49
query93	3044	506	508	506
query94	1101	212	212	212
query95	414	312	305	305
query96	589	278	263	263
query97	3222	3008	3013	3008
query98	226	199	196	196
query99	1180	857	848	848
Total cold run time: 270251 ms
Total hot run time: 172465 ms

doris-robot · 2024-07-01T09:04:40Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19e151152c45885dc3dae2acaed41a6febfb14a9, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.79	0.78	0.79
query15	0.84	0.82	0.80
query16	0.36	0.37	0.37
query17	0.95	0.96	0.95
query18	0.23	0.24	0.26
query19	1.82	1.78	1.72
query20	0.01	0.00	0.00
query21	15.44	0.76	0.65
query22	4.47	7.01	1.80
query23	18.32	1.38	1.35
query24	2.07	0.24	0.23
query25	0.16	0.09	0.09
query26	0.26	0.18	0.17
query27	0.08	0.09	0.09
query28	13.29	1.02	1.00
query29	12.64	3.30	3.29
query30	0.26	0.05	0.06
query31	2.88	0.40	0.37
query32	3.26	0.46	0.48
query33	2.94	2.92	2.90
query34	17.23	4.40	4.40
query35	4.46	4.45	4.52
query36	0.69	0.47	0.47
query37	0.19	0.15	0.15
query38	0.16	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.94 s
Total hot run time: 30.55 s

xzj7019 · 2024-07-01T13:16:41Z

run p0

xzj7019 · 2024-07-02T03:42:54Z

run buildall

doris-robot · 2024-07-02T04:46:50Z

TPC-H: Total hot run time: 39923 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8648516e3113f3caee9224877815e1bced668487, data reload: false

------ Round 1 ----------------------------------
q1	18549	4715	4417	4417
q2	2016	190	195	190
q3	10488	1232	1138	1138
q4	10188	791	786	786
q5	7524	2661	2639	2639
q6	223	135	137	135
q7	956	595	607	595
q8	9228	2120	2078	2078
q9	9008	6536	6556	6536
q10	8953	3746	3780	3746
q11	454	229	231	229
q12	412	232	223	223
q13	18962	2977	2964	2964
q14	277	229	233	229
q15	522	486	461	461
q16	518	392	365	365
q17	984	712	681	681
q18	8184	7444	7421	7421
q19	5453	1442	1323	1323
q20	669	346	320	320
q21	4950	3119	3948	3119
q22	393	350	328	328
Total cold run time: 118911 ms
Total hot run time: 39923 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4251	4264	4251
q2	364	271	263	263
q3	2957	2816	2762	2762
q4	1875	1615	1646	1615
q5	5267	5248	5253	5248
q6	217	131	129	129
q7	2061	1759	1722	1722
q8	3216	3374	3358	3358
q9	8307	8329	8361	8329
q10	3927	3689	3654	3654
q11	582	484	483	483
q12	791	642	590	590
q13	16714	2959	2988	2959
q14	289	257	272	257
q15	522	482	485	482
q16	468	407	406	406
q17	1779	1488	1469	1469
q18	7634	7596	7313	7313
q19	1761	1585	1678	1585
q20	1992	1762	1775	1762
q21	4923	4745	4867	4745
q22	617	507	577	507
Total cold run time: 70682 ms
Total hot run time: 53889 ms

doris-robot · 2024-07-02T04:58:04Z

TPC-DS: Total hot run time: 169117 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8648516e3113f3caee9224877815e1bced668487, data reload: false

query1	916	372	376	372
query2	6467	2312	2280	2280
query3	6654	213	213	213
query4	19148	17609	17263	17263
query5	4192	487	467	467
query6	280	165	186	165
query7	4598	316	286	286
query8	301	280	286	280
query9	8450	2376	2369	2369
query10	616	306	304	304
query11	10680	10223	9952	9952
query12	139	87	83	83
query13	1644	365	358	358
query14	10204	6777	7672	6777
query15	238	183	186	183
query16	7927	327	321	321
query17	1792	551	545	545
query18	1983	276	283	276
query19	196	152	158	152
query20	94	86	86	86
query21	216	134	127	127
query22	4325	4038	3932	3932
query23	33542	33173	33027	33027
query24	11770	2774	2802	2774
query25	653	385	379	379
query26	1570	156	157	156
query27	2918	312	322	312
query28	7604	2077	2059	2059
query29	1003	661	635	635
query30	281	155	165	155
query31	978	724	772	724
query32	96	57	62	57
query33	792	306	304	304
query34	923	461	467	461
query35	750	632	641	632
query36	1117	948	894	894
query37	159	76	77	76
query38	2868	2760	2762	2760
query39	845	798	811	798
query40	287	129	129	129
query41	57	54	56	54
query42	137	104	101	101
query43	569	517	539	517
query44	1199	723	721	721
query45	191	169	171	169
query46	1085	744	735	735
query47	1846	1741	1768	1741
query48	372	323	313	313
query49	1182	421	428	421
query50	772	388	384	384
query51	6870	6859	6901	6859
query52	105	89	96	89
query53	358	308	286	286
query54	1030	454	447	447
query55	78	78	74	74
query56	297	273	293	273
query57	1162	1031	1080	1031
query58	265	250	379	250
query59	3355	3122	3116	3116
query60	315	287	275	275
query61	92	91	97	91
query62	659	453	444	444
query63	318	288	290	288
query64	9942	2228	1777	1777
query65	3172	3172	3126	3126
query66	1369	327	323	323
query67	15735	15126	14906	14906
query68	6973	550	571	550
query69	708	506	348	348
query70	1238	1118	1030	1030
query71	508	270	270	270
query72	9238	2788	2630	2630
query73	821	323	322	322
query74	5943	5553	5582	5553
query75	4997	2633	2678	2633
query76	4919	967	948	948
query77	776	306	301	301
query78	9571	9157	8907	8907
query79	8264	523	541	523
query80	1001	472	465	465
query81	570	213	226	213
query82	815	104	108	104
query83	369	165	167	165
query84	275	86	90	86
query85	1399	281	268	268
query86	454	315	305	305
query87	3278	3125	3135	3125
query88	4840	2366	2383	2366
query89	543	382	378	378
query90	2004	183	185	183
query91	128	99	97	97
query92	67	49	50	49
query93	7041	501	502	501
query94	1265	209	213	209
query95	398	313	314	313
query96	612	265	272	265
query97	3173	3038	3045	3038
query98	220	204	196	196
query99	1168	842	850	842
Total cold run time: 295324 ms
Total hot run time: 169117 ms

doris-robot · 2024-07-02T05:03:35Z

ClickBench: Total hot run time: 30.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8648516e3113f3caee9224877815e1bced668487, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.55	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.47
query10	0.54	0.53	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.58	0.59	0.59
query14	0.76	0.78	0.78
query15	0.84	0.83	0.82
query16	0.37	0.37	0.37
query17	0.95	0.94	1.03
query18	0.24	0.22	0.25
query19	1.76	1.69	1.66
query20	0.01	0.01	0.01
query21	15.44	0.78	0.67
query22	4.70	6.70	2.07
query23	18.27	1.38	1.31
query24	2.16	0.23	0.23
query25	0.15	0.09	0.08
query26	0.28	0.17	0.17
query27	0.07	0.08	0.08
query28	13.20	1.02	1.01
query29	12.65	3.42	3.41
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.25	0.48	0.46
query33	2.93	2.86	2.85
query34	17.07	4.37	4.43
query35	4.49	4.48	4.51
query36	0.64	0.46	0.46
query37	0.19	0.15	0.17
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.03	0.04
Total cold run time: 109.82 s
Total hot run time: 30.76 s

github-actions · 2024-07-02T09:21:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-02T09:21:04Z

PR approved by anyone and no changes requested.

mrhhsg

LGTM

xzj7019 · 2024-07-03T07:01:55Z

run buildall

doris-robot · 2024-07-03T07:38:40Z

TPC-H: Total hot run time: 39834 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 622dc814f78031a15710180cdeb9cbf7bceca1da, data reload: false

------ Round 1 ----------------------------------
q1	17614	4380	4330	4330
q2	2009	187	181	181
q3	10473	1214	1065	1065
q4	10197	756	773	756
q5	7507	2624	2583	2583
q6	222	138	135	135
q7	957	586	598	586
q8	9229	2083	2058	2058
q9	9008	6472	6442	6442
q10	8995	3773	3813	3773
q11	473	247	248	247
q12	474	229	230	229
q13	17783	3005	2998	2998
q14	278	227	221	221
q15	509	479	500	479
q16	494	377	375	375
q17	969	646	663	646
q18	7989	7449	7415	7415
q19	3692	1453	1543	1453
q20	647	329	316	316
q21	4948	3769	3220	3220
q22	407	329	326	326
Total cold run time: 114874 ms
Total hot run time: 39834 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4393	4331	4283	4283
q2	370	260	257	257
q3	2938	2717	3121	2717
q4	2024	1757	1731	1731
q5	5574	5536	5482	5482
q6	235	138	129	129
q7	2172	1840	1850	1840
q8	3271	3396	3411	3396
q9	8658	8570	8730	8570
q10	4111	3912	3873	3873
q11	583	512	478	478
q12	778	643	670	643
q13	16061	3192	3163	3163
q14	302	294	267	267
q15	545	498	475	475
q16	479	423	438	423
q17	1803	1522	1464	1464
q18	8093	7871	7788	7788
q19	3248	1679	1612	1612
q20	2228	1826	1860	1826
q21	5130	4907	4841	4841
q22	604	522	524	522
Total cold run time: 73600 ms
Total hot run time: 55780 ms

doris-robot · 2024-07-03T07:49:35Z

TPC-DS: Total hot run time: 169884 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 622dc814f78031a15710180cdeb9cbf7bceca1da, data reload: false

query1	932	387	374	374
query2	6465	2422	2301	2301
query3	6637	205	219	205
query4	18809	17464	17160	17160
query5	3607	488	489	488
query6	262	174	167	167
query7	4587	292	288	288
query8	330	280	275	275
query9	8583	2353	2335	2335
query10	560	275	278	275
query11	10488	10232	10000	10000
query12	109	90	82	82
query13	1646	373	353	353
query14	10119	7170	7603	7170
query15	245	187	184	184
query16	7737	309	293	293
query17	1814	530	521	521
query18	1951	275	269	269
query19	199	151	146	146
query20	90	80	81	80
query21	211	125	124	124
query22	4360	3961	4076	3961
query23	33710	33564	33467	33467
query24	10975	2809	2799	2799
query25	652	391	365	365
query26	919	152	149	149
query27	2331	330	323	323
query28	6438	2141	2116	2116
query29	863	651	634	634
query30	230	157	153	153
query31	968	746	752	746
query32	103	57	57	57
query33	740	282	300	282
query34	991	466	475	466
query35	737	629	640	629
query36	1124	972	960	960
query37	145	74	89	74
query38	2917	2835	2820	2820
query39	950	843	846	843
query40	209	124	122	122
query41	53	50	54	50
query42	107	98	104	98
query43	579	575	542	542
query44	1211	752	728	728
query45	186	164	162	162
query46	1084	726	727	726
query47	1879	1761	1795	1761
query48	380	308	313	308
query49	844	401	409	401
query50	772	378	381	378
query51	6902	6726	6760	6726
query52	98	97	89	89
query53	360	291	281	281
query54	897	453	444	444
query55	72	71	74	71
query56	281	272	303	272
query57	1114	1058	1065	1058
query58	250	247	245	245
query59	3258	3308	3073	3073
query60	300	281	271	271
query61	96	92	92	92
query62	613	450	455	450
query63	321	287	285	285
query64	8514	2313	1858	1858
query65	3162	3072	3099	3072
query66	774	333	357	333
query67	15253	15010	14896	14896
query68	4473	537	542	537
query69	522	424	321	321
query70	1201	1156	1087	1087
query71	387	289	284	284
query72	7072	2956	2766	2766
query73	748	328	331	328
query74	5785	5561	5435	5435
query75	3431	2684	2659	2659
query76	2820	933	909	909
query77	440	314	305	305
query78	9580	8890	8903	8890
query79	3819	512	517	512
query80	2856	486	490	486
query81	585	228	224	224
query82	977	105	108	105
query83	314	181	177	177
query84	278	92	107	92
query85	1997	462	264	264
query86	484	274	323	274
query87	3288	3104	3056	3056
query88	4419	2436	2437	2436
query89	501	365	376	365
query90	1866	190	184	184
query91	126	96	100	96
query92	67	49	49	49
query93	5251	508	502	502
query94	1214	207	270	207
query95	403	307	313	307
query96	588	273	265	265
query97	3134	3004	3029	3004
query98	221	207	195	195
query99	1220	847	837	837
Total cold run time: 274170 ms
Total hot run time: 169884 ms

doris-robot · 2024-07-03T07:55:07Z

ClickBench: Total hot run time: 30.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 622dc814f78031a15710180cdeb9cbf7bceca1da, data reload: false

query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.65	0.07	0.06
query5	0.46	0.50	0.49
query6	1.13	0.72	0.70
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.51
query10	0.55	0.55	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.78	0.78
query15	0.85	0.81	0.82
query16	0.36	0.35	0.36
query17	0.97	1.02	1.05
query18	0.23	0.25	0.24
query19	1.87	1.83	1.82
query20	0.02	0.01	0.01
query21	15.43	0.76	0.66
query22	3.49	7.93	1.78
query23	18.23	1.41	1.23
query24	2.13	0.25	0.22
query25	0.16	0.10	0.08
query26	0.27	0.17	0.17
query27	0.09	0.08	0.08
query28	13.23	1.03	0.99
query29	12.58	3.21	3.21
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.30	0.46	0.46
query33	2.90	2.84	2.90
query34	17.17	4.43	4.43
query35	4.47	4.48	4.54
query36	0.65	0.45	0.46
query37	0.20	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.10	0.04	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.69 s
Total hot run time: 30.38 s

github-actions · 2024-07-03T11:22:54Z

PR approved by at least one committer and no changes requested.

mrhhsg

lgtm

Current left semi/anti join's cost doesn't consider be's short-cut opt, this pr will refine this part of computing and distinguish the left and right join costs. --------- Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>

Refine left semi/anti cost computing under short-cut opt, for the case whose semi/anti join has the small left side and big right side, which original solution can't support. This pr reduce the left style cost by reduce the right side cost and improve the possibility of choosing left style joins. Pass the performance test on tpch/tpcds/usercase. previous work: #37060

xzj7019 marked this pull request as draft July 1, 2024 04:32

xzj7019 marked this pull request as ready for review July 1, 2024 06:40

xzj7019 force-pushed the semi_anti_cost_refine branch from 19e1511 to 8648516 Compare July 2, 2024 03:42

morrySnow previously approved these changes Jul 2, 2024

View reviewed changes

morrySnow added the dev/2.1.x label Jul 2, 2024

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 2, 2024

mrhhsg previously approved these changes Jul 3, 2024

View reviewed changes

[opt](nereids) refine left semi/anti shortcut cost

80fba89

[opt](nereids) refine left semi/anti shortcut cost

622dc81

xzj7019 dismissed stale reviews from mrhhsg and morrySnow via 622dc81 July 3, 2024 07:01

xzj7019 force-pushed the semi_anti_cost_refine branch from 8648516 to 622dc81 Compare July 3, 2024 07:01

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

morrySnow approved these changes Jul 3, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

mrhhsg approved these changes Jul 3, 2024

View reviewed changes

morrySnow merged commit d81ed7b into apache:master Jul 3, 2024

morrySnow removed the dev/2.1.x label Jul 17, 2024

morrySnow mentioned this pull request Jul 17, 2024

[opt](nereids) refine left semi/anti cost under short-cut opt #37951

Merged

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Conversation

xzj7019 commented Jul 1, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 1, 2024

Uh oh!

xzj7019 commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

xzj7019 commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

github-actions bot commented Jul 3, 2024

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants