Skip to content

[fix](oracle scan) Fix performance issues caused by version judgment#41407

Merged
zy-kkk merged 1 commit intoapache:masterfrom
zy-kkk:fix_oracle_scanner_performance
Sep 30, 2024
Merged

[fix](oracle scan) Fix performance issues caused by version judgment#41407
zy-kkk merged 1 commit intoapache:masterfrom
zy-kkk:fix_oracle_scanner_performance

Conversation

@zy-kkk
Copy link
Copy Markdown
Member

@zy-kkk zy-kkk commented Sep 27, 2024

No description provided.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Copy Markdown
Member Author

zy-kkk commented Sep 27, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 27, 2024

TPC-H: Total hot run time: 41024 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d77110a4494f911bf2b2e45f1feac57b44c74695, data reload: false

------ Round 1 ----------------------------------
q1	17916	7420	7397	7397
q2	2228	180	156	156
q3	11655	1159	1192	1159
q4	11078	750	754	750
q5	7805	2915	2881	2881
q6	233	153	156	153
q7	1004	646	613	613
q8	9505	1892	2000	1892
q9	6531	6380	6384	6380
q10	6938	2263	2302	2263
q11	447	243	251	243
q12	409	218	220	218
q13	17755	2987	2997	2987
q14	234	208	214	208
q15	563	516	505	505
q16	632	580	601	580
q17	971	525	518	518
q18	7292	6660	6697	6660
q19	1395	1083	1056	1056
q20	492	205	205	205
q21	4036	3188	3268	3188
q22	1085	1013	1012	1012
Total cold run time: 110204 ms
Total hot run time: 41024 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7234	7212	7337	7212
q2	333	226	225	225
q3	2883	2699	2794	2699
q4	1948	1685	1767	1685
q5	5367	5454	5454	5454
q6	226	140	140	140
q7	2139	1689	1712	1689
q8	3202	3387	3384	3384
q9	8513	8523	8490	8490
q10	3464	3450	3442	3442
q11	577	486	476	476
q12	807	580	607	580
q13	5654	2976	2971	2971
q14	282	263	259	259
q15	552	521	512	512
q16	665	633	644	633
q17	1790	1560	1564	1560
q18	7652	7323	7491	7323
q19	1649	1498	1518	1498
q20	2053	1813	1835	1813
q21	5495	5142	5052	5052
q22	1116	1011	991	991
Total cold run time: 63601 ms
Total hot run time: 58088 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 27, 2024

TPC-DS: Total hot run time: 190638 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d77110a4494f911bf2b2e45f1feac57b44c74695, data reload: false

query1	1009	375	376	375
query2	6515	2087	2021	2021
query3	6707	213	224	213
query4	33598	23295	23213	23213
query5	4315	453	482	453
query6	254	164	158	158
query7	4614	307	312	307
query8	273	226	227	226
query9	9647	2633	2626	2626
query10	474	298	286	286
query11	17812	15199	15101	15101
query12	149	95	99	95
query13	1655	422	414	414
query14	10367	7044	7299	7044
query15	284	177	176	176
query16	8151	468	471	468
query17	1666	575	555	555
query18	2149	298	311	298
query19	363	150	148	148
query20	115	108	106	106
query21	220	105	104	104
query22	4900	4652	4571	4571
query23	34758	33910	33860	33860
query24	11242	2741	2890	2741
query25	672	408	411	408
query26	1418	165	159	159
query27	2857	294	297	294
query28	8037	2428	2416	2416
query29	942	439	430	430
query30	319	165	156	156
query31	1014	802	798	798
query32	101	55	58	55
query33	770	303	301	301
query34	963	499	487	487
query35	891	728	745	728
query36	1098	939	906	906
query37	151	88	87	87
query38	3978	3865	3885	3865
query39	1466	1425	1450	1425
query40	289	98	97	97
query41	52	46	50	46
query42	115	99	97	97
query43	541	503	493	493
query44	1257	827	781	781
query45	199	167	166	166
query46	1141	725	721	721
query47	1966	1857	1887	1857
query48	448	369	370	369
query49	1148	415	409	409
query50	847	404	404	404
query51	7197	6896	6978	6896
query52	100	87	86	86
query53	268	195	178	178
query54	1241	466	469	466
query55	81	76	81	76
query56	277	265	265	265
query57	1273	1075	1123	1075
query58	256	236	232	232
query59	3311	2895	2936	2895
query60	323	272	267	267
query61	163	99	101	99
query62	862	660	647	647
query63	224	191	185	185
query64	5327	643	600	600
query65	3364	3346	3192	3192
query66	1432	306	301	301
query67	15943	15593	15492	15492
query68	4665	558	555	555
query69	456	295	316	295
query70	1149	1110	1122	1110
query71	339	269	267	267
query72	6900	3912	4072	3912
query73	757	336	338	336
query74	9892	8922	8933	8922
query75	3411	2660	2673	2660
query76	2964	893	883	883
query77	409	283	293	283
query78	10597	9606	9551	9551
query79	1388	574	604	574
query80	1228	443	444	443
query81	585	242	238	238
query82	965	139	138	138
query83	207	133	142	133
query84	255	81	76	76
query85	1263	282	276	276
query86	361	307	297	297
query87	4479	4259	4280	4259
query88	2998	2382	2348	2348
query89	402	282	283	282
query90	1828	187	185	185
query91	176	143	141	141
query92	58	48	46	46
query93	1061	543	538	538
query94	863	291	286	286
query95	339	245	250	245
query96	627	276	274	274
query97	3252	3145	3133	3133
query98	230	196	192	192
query99	1549	1294	1302	1294
Total cold run time: 301212 ms
Total hot run time: 190638 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 27, 2024

ClickBench: Total hot run time: 32.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d77110a4494f911bf2b2e45f1feac57b44c74695, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.07
query4	1.65	0.10	0.09
query5	0.50	0.50	0.52
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.55	0.57	0.54
query11	0.13	0.11	0.12
query12	0.14	0.11	0.10
query13	0.60	0.60	0.59
query14	2.68	2.68	2.72
query15	0.90	0.82	0.82
query16	0.37	0.38	0.38
query17	0.98	1.05	1.06
query18	0.22	0.22	0.21
query19	1.99	1.89	2.04
query20	0.01	0.01	0.02
query21	15.40	0.60	0.60
query22	2.38	2.48	1.87
query23	17.06	1.06	0.94
query24	2.75	0.79	0.76
query25	0.27	0.18	0.11
query26	0.36	0.14	0.13
query27	0.04	0.04	0.04
query28	11.30	1.09	1.06
query29	12.56	3.30	3.30
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.30	0.46	0.46
query33	3.02	3.03	3.03
query34	16.91	4.46	4.38
query35	4.52	4.47	4.50
query36	0.67	0.49	0.47
query37	0.07	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.91 s
Total hot run time: 32.72 s

morningman
morningman previously approved these changes Sep 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
Copy link
Copy Markdown
Contributor

morningman commented Sep 29, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 29, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 29, 2024

PR approved by anyone and no changes requested.

CalvinKirs
CalvinKirs previously approved these changes Sep 29, 2024
View reviewed changes
@morningman morningman force-pushed the fix_oracle_scanner_performance branch from a5ef3a2 to d4f234d Compare September 30, 2024 07:54
@morningman
Copy link
Copy Markdown
Contributor

morningman commented Sep 30, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 30, 2024

TPC-H: Total hot run time: 40727 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4f234d5e5b266f157c6e2caedf4c3bc8d65a088, data reload: false

------ Round 1 ----------------------------------
q1	17771	7483	7237	7237
q2	2025	277	265	265
q3	11993	1071	1209	1071
q4	10569	758	720	720
q5	7769	2893	2874	2874
q6	239	151	147	147
q7	997	634	621	621
q8	9364	1989	1940	1940
q9	6580	6411	6403	6403
q10	6985	2293	2300	2293
q11	444	245	245	245
q12	421	217	213	213
q13	17773	2995	2990	2990
q14	248	210	221	210
q15	569	523	514	514
q16	638	585	583	583
q17	968	517	577	517
q18	7454	6631	6717	6631
q19	1336	944	944	944
q20	498	212	207	207
q21	4011	3116	3239	3116
q22	1116	986	989	986
Total cold run time: 109768 ms
Total hot run time: 40727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7240	7211	7220	7211
q2	336	229	229	229
q3	3027	2912	2900	2900
q4	2124	1926	1815	1815
q5	5814	5749	5761	5749
q6	232	150	143	143
q7	2266	1845	1854	1845
q8	3359	3545	3421	3421
q9	8944	8926	8826	8826
q10	3585	3557	3503	3503
q11	573	491	475	475
q12	813	661	652	652
q13	9602	3182	3157	3157
q14	312	269	293	269
q15	573	526	530	526
q16	701	636	653	636
q17	1851	1625	1599	1599
q18	8361	7817	7489	7489
q19	1719	1537	1600	1537
q20	2104	1888	1881	1881
q21	5454	5358	5430	5358
q22	1120	1031	1055	1031
Total cold run time: 70110 ms
Total hot run time: 60252 ms

@zy-kkk zy-kkk dismissed stale reviews from CalvinKirs and morningman via 73f61d7 September 30, 2024 09:46
@zy-kkk zy-kkk force-pushed the fix_oracle_scanner_performance branch from d4f234d to 73f61d7 Compare September 30, 2024 09:46
@zy-kkk
Copy link
Copy Markdown
Member Author

zy-kkk commented Sep 30, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 30, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 30, 2024

TPC-H: Total hot run time: 40776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73f61d723594a670628d5f35e0b4a29761cb1b6a, data reload: false

------ Round 1 ----------------------------------
q1	17555	7328	7262	7262
q2	2041	158	175	158
q3	10548	1147	1155	1147
q4	10212	780	697	697
q5	7759	2921	2823	2823
q6	244	148	153	148
q7	1010	633	609	609
q8	9571	2048	1986	1986
q9	8022	6393	6391	6391
q10	7052	2279	2311	2279
q11	442	242	251	242
q12	529	219	215	215
q13	17804	2978	3004	2978
q14	237	204	214	204
q15	571	517	519	517
q16	690	580	593	580
q17	1005	573	539	539
q18	7276	6719	6761	6719
q19	1344	1003	910	910
q20	485	207	203	203
q21	3990	3191	3198	3191
q22	1103	978	984	978
Total cold run time: 109490 ms
Total hot run time: 40776 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7229	7202	7201	7201
q2	339	227	225	225
q3	2916	2737	2797	2737
q4	1970	1813	1714	1714
q5	5445	5459	5479	5459
q6	225	141	138	138
q7	2151	1681	1695	1681
q8	3223	3399	3440	3399
q9	8535	8525	8493	8493
q10	3469	3424	3396	3396
q11	575	465	470	465
q12	799	561	620	561
q13	3727	3008	2949	2949
q14	300	265	264	264
q15	564	509	515	509
q16	666	655	634	634
q17	1824	1572	1536	1536
q18	7757	7376	7550	7376
q19	1657	1583	1572	1572
q20	2057	1808	1800	1800
q21	5394	5326	5146	5146
q22	1098	1024	967	967
Total cold run time: 61920 ms
Total hot run time: 58222 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 30, 2024

TPC-DS: Total hot run time: 192195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73f61d723594a670628d5f35e0b4a29761cb1b6a, data reload: false

query1	940	372	361	361
query2	6187	2058	2050	2050
query3	6208	210	228	210
query4	33360	23582	23349	23349
query5	3916	463	468	463
query6	256	171	154	154
query7	4450	323	320	320
query8	276	232	211	211
query9	9277	2636	2660	2636
query10	487	289	280	280
query11	18063	15140	15054	15054
query12	147	97	95	95
query13	1633	465	439	439
query14	9753	7467	7540	7467
query15	300	171	172	171
query16	7887	493	453	453
query17	1732	576	575	575
query18	2106	310	307	307
query19	356	153	151	151
query20	120	107	108	107
query21	214	103	100	100
query22	4636	4389	4168	4168
query23	34936	34192	34408	34192
query24	10723	2893	2874	2874
query25	621	399	426	399
query26	1156	163	164	163
query27	2218	300	297	297
query28	7190	2430	2433	2430
query29	836	436	442	436
query30	328	157	157	157
query31	1075	817	827	817
query32	99	56	53	53
query33	772	292	292	292
query34	925	505	527	505
query35	885	754	754	754
query36	1096	977	963	963
query37	153	86	84	84
query38	4178	3879	3886	3879
query39	1501	1422	1425	1422
query40	204	100	97	97
query41	48	45	45	45
query42	115	96	103	96
query43	538	497	501	497
query44	1255	819	812	812
query45	198	169	171	169
query46	1155	739	720	720
query47	1956	1781	1814	1781
query48	442	358	351	351
query49	1092	414	405	405
query50	815	411	426	411
query51	7086	7035	7006	7006
query52	101	87	86	86
query53	260	190	185	185
query54	1201	477	479	477
query55	82	79	78	78
query56	295	272	270	270
query57	1242	1177	1150	1150
query58	247	281	232	232
query59	3233	3048	2940	2940
query60	297	288	305	288
query61	100	105	102	102
query62	866	667	655	655
query63	218	194	189	189
query64	3932	628	599	599
query65	3265	3227	3199	3199
query66	833	308	311	308
query67	15916	15634	15695	15634
query68	3978	567	577	567
query69	614	300	319	300
query70	1160	1140	1152	1140
query71	379	276	283	276
query72	7477	4153	3968	3968
query73	769	352	356	352
query74	10210	8872	9097	8872
query75	3497	2655	2668	2655
query76	3090	870	878	870
query77	646	302	308	302
query78	10692	9698	9681	9681
query79	2499	590	617	590
query80	1128	432	441	432
query81	574	240	247	240
query82	706	142	133	133
query83	312	132	130	130
query84	275	82	77	77
query85	1351	298	279	279
query86	457	304	296	296
query87	4458	4315	4333	4315
query88	3621	2545	2408	2408
query89	395	284	289	284
query90	2101	189	191	189
query91	141	105	103	103
query92	75	45	45	45
query93	1542	554	554	554
query94	1095	298	297	297
query95	360	256	258	256
query96	617	277	286	277
query97	3308	3202	3195	3195
query98	217	198	186	186
query99	1536	1306	1314	1306
Total cold run time: 297268 ms
Total hot run time: 192195 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 30, 2024

ClickBench: Total hot run time: 32.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 73f61d723594a670628d5f35e0b4a29761cb1b6a, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.50	0.51
query6	1.15	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.54	0.56	0.55
query11	0.13	0.10	0.11
query12	0.15	0.11	0.10
query13	0.60	0.60	0.61
query14	2.72	2.74	2.71
query15	0.91	0.83	0.82
query16	0.38	0.39	0.38
query17	1.09	1.05	1.02
query18	0.19	0.19	0.20
query19	1.92	1.83	2.00
query20	0.01	0.01	0.02
query21	15.36	0.61	0.58
query22	2.74	1.59	2.24
query23	16.98	1.19	0.76
query24	2.92	1.17	0.80
query25	0.21	0.09	0.06
query26	0.55	0.14	0.13
query27	0.05	0.06	0.05
query28	10.97	1.10	1.07
query29	12.56	3.22	3.21
query30	0.25	0.06	0.05
query31	2.88	0.38	0.39
query32	3.27	0.48	0.47
query33	2.99	3.04	3.04
query34	17.11	4.52	4.45
query35	4.48	4.47	4.51
query36	0.65	0.47	0.49
query37	0.08	0.05	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.36 s
Total hot run time: 32.17 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 30, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 30, 2024

PR approved by at least one committer and no changes requested.

@zy-kkk zy-kkk merged commit f9bd4ef into apache:master Sep 30, 2024
@zy-kkk zy-kkk deleted the fix_oracle_scanner_performance branch September 30, 2024 14:32
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 30, 2024
@zy-kkk
[fix](oracle scan) Fix performance issues caused by version judgment (a…
f0a9ca4 
…pache#41407)
@zy-kkk zy-kkk mentioned this pull request Sep 30, 2024
Merged
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 30, 2024
@zy-kkk
[fix](oracle scan) Fix performance issues caused by version judgment (a…
6bfe61c 
…pache#41407)
@zy-kkk zy-kkk mentioned this pull request Sep 30, 2024
Merged
morningman pushed a commit that referenced this pull request Oct 1, 2024
@zy-kkk
[2.1][fix](oracle scan) Fix performance issues caused by version judg…
806fe98 
…ment (#41509)

pick (#41407)
morningman pushed a commit that referenced this pull request Oct 1, 2024
@zy-kkk
[3.0][fix](oracle scan) Fix performance issues caused by version judg…
d61a23d 
…ment (#41513)

pick (#41407)
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Oct 10, 2024
@zy-kkk @eldenmoon
[fix](oracle scan) Fix performance issues caused by version judgment (a…
797668c 
…pache#41407)
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@zy-kkk @cjj2010
[fix](oracle scan) Fix performance issues caused by version judgment (a…
d9e42df 
…pache#41407)
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
@zy-kkk
[fix](oracle scan) Fix performance issues caused by version judgment (a…
10659b2 
…pache#41407)
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Closed
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@CalvinKirs CalvinKirs CalvinKirs left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zy-kkk @doris-robot @morningman @CalvinKirs @wuwenchi