Skip to content

[improve](udaf)support class cache for java-udaf#47619

Merged
zhangstar333 merged 5 commits intoapache:masterfrom
zhangstar333:java_cache
Feb 13, 2025
Merged

[improve](udaf)support class cache for java-udaf#47619
zhangstar333 merged 5 commits intoapache:masterfrom
zhangstar333:java_cache

Conversation

@zhangstar333
Copy link
Copy Markdown
Contributor

@zhangstar333 zhangstar333 commented Feb 8, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:
in some user case,

  • the udf-jars package have some resource file, it's maybe about hundreds MBs,
    so if every instances load the jar, it's easy to cause the BE JVM OOM.
  • or in some udf-jars have some init operator cause many times, so wants all instance could only init it's once.

follow up #40404
support class cache for java-udaf

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Feb 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333
Copy link
Copy Markdown
Contributor Author

zhangstar333 commented Feb 8, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 8, 2025

TPC-H: Total hot run time: 31482 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

------ Round 1 ----------------------------------
q1	17589	5136	5072	5072
q2	2043	315	170	170
q3	10626	1223	765	765
q4	10254	1028	552	552
q5	7898	2370	2402	2370
q6	188	168	133	133
q7	892	765	607	607
q8	9282	1285	1074	1074
q9	4828	4772	4534	4534
q10	6824	2336	1900	1900
q11	462	283	255	255
q12	342	347	213	213
q13	17748	3658	3074	3074
q14	236	219	212	212
q15	526	486	475	475
q16	632	608	577	577
q17	555	868	328	328
q18	6739	6201	6285	6201
q19	1211	956	540	540
q20	310	317	181	181
q21	2666	2158	1943	1943
q22	376	327	306	306
Total cold run time: 102227 ms
Total hot run time: 31482 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5115	5105	5126	5105
q2	245	327	230	230
q3	2141	2694	2319	2319
q4	1475	1864	1438	1438
q5	4234	4161	4113	4113
q6	210	167	123	123
q7	1872	1801	1684	1684
q8	2612	2685	2586	2586
q9	7109	7130	7219	7130
q10	3043	3257	2800	2800
q11	582	528	508	508
q12	681	772	620	620
q13	3577	3942	3355	3355
q14	293	283	261	261
q15	512	476	466	466
q16	637	669	660	660
q17	1137	1602	1332	1332
q18	7506	7272	7275	7272
q19	777	786	955	786
q20	1997	2035	1878	1878
q21	5434	4968	4784	4784
q22	652	581	545	545
Total cold run time: 51841 ms
Total hot run time: 49995 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 8, 2025

TPC-DS: Total hot run time: 189497 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

query1	1329	962	933	933
query2	6238	1849	1889	1849
query3	11090	4638	4666	4638
query4	55010	25676	22957	22957
query5	4919	550	473	473
query6	332	198	187	187
query7	4901	517	298	298
query8	307	251	241	241
query9	5622	2577	2588	2577
query10	430	304	271	271
query11	15171	15034	14798	14798
query12	154	106	109	106
query13	1065	504	397	397
query14	10201	6941	6260	6260
query15	212	192	182	182
query16	7072	681	518	518
query17	1070	748	578	578
query18	1541	414	317	317
query19	203	199	173	173
query20	127	135	121	121
query21	212	130	109	109
query22	4446	4471	4472	4471
query23	34276	33417	33279	33279
query24	5736	2455	2413	2413
query25	457	455	412	412
query26	705	283	156	156
query27	1877	495	327	327
query28	2953	2438	2405	2405
query29	575	562	433	433
query30	217	196	158	158
query31	885	879	810	810
query32	71	69	62	62
query33	443	349	310	310
query34	870	871	493	493
query35	795	837	757	757
query36	931	998	919	919
query37	125	106	81	81
query38	4248	4331	4271	4271
query39	1489	1467	1439	1439
query40	205	118	103	103
query41	52	52	48	48
query42	121	100	101	100
query43	513	508	504	504
query44	1295	833	800	800
query45	177	175	167	167
query46	870	1056	652	652
query47	1883	1912	1807	1807
query48	386	407	303	303
query49	709	539	414	414
query50	696	764	414	414
query51	4275	4374	4223	4223
query52	110	109	95	95
query53	231	268	189	189
query54	474	498	408	408
query55	82	82	76	76
query56	279	283	277	277
query57	1170	1208	1134	1134
query58	253	247	237	237
query59	2773	3022	2763	2763
query60	282	286	266	266
query61	116	118	118	118
query62	792	762	674	674
query63	233	187	182	182
query64	1678	1028	693	693
query65	3246	3127	3153	3127
query66	759	389	296	296
query67	16083	15580	15431	15431
query68	4864	770	512	512
query69	508	299	274	274
query70	1205	1134	1103	1103
query71	434	290	260	260
query72	5918	3609	3752	3609
query73	1164	744	350	350
query74	8948	8978	8754	8754
query75	3184	3144	2697	2697
query76	3724	1178	733	733
query77	563	354	270	270
query78	9991	9996	9296	9296
query79	1807	811	574	574
query80	653	520	447	447
query81	493	277	236	236
query82	276	146	121	121
query83	176	166	150	150
query84	344	89	73	73
query85	732	348	316	316
query86	335	296	301	296
query87	4462	4482	4346	4346
query88	2794	2176	2151	2151
query89	392	308	275	275
query90	1805	190	189	189
query91	137	141	108	108
query92	75	61	55	55
query93	2109	1012	574	574
query94	672	409	298	298
query95	342	261	251	251
query96	477	552	273	273
query97	2758	2821	2754	2754
query98	222	202	195	195
query99	1304	1398	1277	1277
Total cold run time: 290916 ms
Total hot run time: 189497 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 8, 2025

ClickBench: Total hot run time: 30.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.42	0.40
query6	1.17	0.68	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.59	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.62	0.61
query14	2.67	2.71	2.83
query15	0.92	0.85	0.86
query16	0.38	0.39	0.37
query17	1.02	1.07	1.06
query18	0.22	0.20	0.20
query19	1.88	1.75	2.02
query20	0.02	0.01	0.02
query21	15.39	0.89	0.54
query22	0.75	1.16	0.62
query23	14.97	1.41	0.62
query24	7.83	1.75	0.48
query25	0.50	0.26	0.09
query26	0.56	0.16	0.14
query27	0.05	0.05	0.05
query28	9.20	0.85	0.42
query29	12.56	4.00	3.28
query30	0.25	0.09	0.07
query31	2.83	0.59	0.38
query32	3.24	0.54	0.46
query33	3.05	3.01	3.03
query34	15.74	5.16	4.52
query35	4.55	4.58	4.55
query36	0.65	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.38 s
Total hot run time: 30.23 s

@zhangstar333 zhangstar333 marked this pull request as ready for review February 10, 2025 02:00
@zhangstar333
Copy link
Copy Markdown
Contributor Author

zhangstar333 commented Feb 10, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

TPC-H: Total hot run time: 31580 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

------ Round 1 ----------------------------------
q1	17597	5680	5141	5141
q2	2042	304	166	166
q3	10423	1277	725	725
q4	10251	1049	536	536
q5	7887	2331	2390	2331
q6	196	168	130	130
q7	885	754	600	600
q8	9294	1399	1207	1207
q9	4877	4506	4613	4506
q10	6850	2326	1885	1885
q11	474	271	254	254
q12	344	352	221	221
q13	17764	3785	3077	3077
q14	230	237	212	212
q15	505	475	445	445
q16	616	611	589	589
q17	578	916	333	333
q18	6669	6303	6158	6158
q19	1225	954	585	585
q20	327	347	193	193
q21	2957	2217	1969	1969
q22	372	334	317	317
Total cold run time: 102363 ms
Total hot run time: 31580 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5291	5309	5287	5287
q2	247	334	241	241
q3	2202	2805	2430	2430
q4	1517	1920	1400	1400
q5	4223	4129	4159	4129
q6	218	165	126	126
q7	1880	1846	1772	1772
q8	2630	2534	2641	2534
q9	7263	7142	7126	7126
q10	3023	3225	2804	2804
q11	582	509	490	490
q12	700	814	637	637
q13	3430	3977	3269	3269
q14	278	298	273	273
q15	514	474	448	448
q16	655	700	644	644
q17	1180	1570	1358	1358
q18	7471	7257	7310	7257
q19	796	800	897	800
q20	1994	2071	1899	1899
q21	5634	5039	4685	4685
q22	605	578	597	578
Total cold run time: 52333 ms
Total hot run time: 50187 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

TPC-DS: Total hot run time: 190340 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

query1	1303	958	918	918
query2	6095	1899	1919	1899
query3	11110	4698	4453	4453
query4	57371	25900	23687	23687
query5	5045	514	481	481
query6	328	204	188	188
query7	4947	513	292	292
query8	303	254	226	226
query9	5755	2455	2486	2455
query10	410	317	258	258
query11	15152	15079	14890	14890
query12	162	107	112	107
query13	1061	524	407	407
query14	11099	6984	6551	6551
query15	204	194	189	189
query16	7092	665	495	495
query17	1069	742	608	608
query18	1643	433	334	334
query19	221	204	168	168
query20	140	124	124	124
query21	204	128	109	109
query22	4532	4531	4347	4347
query23	33975	33557	33346	33346
query24	5689	2431	2413	2413
query25	460	459	402	402
query26	674	270	155	155
query27	1770	495	334	334
query28	2709	2439	2405	2405
query29	577	567	424	424
query30	237	187	155	155
query31	920	857	808	808
query32	72	67	60	60
query33	456	367	298	298
query34	765	873	502	502
query35	812	825	739	739
query36	953	982	921	921
query37	133	106	73	73
query38	4327	4380	4244	4244
query39	1514	1441	1432	1432
query40	218	116	101	101
query41	52	52	54	52
query42	125	105	112	105
query43	506	546	508	508
query44	1311	835	813	813
query45	182	173	169	169
query46	915	1090	653	653
query47	1827	1914	1809	1809
query48	383	450	325	325
query49	678	494	394	394
query50	702	760	429	429
query51	4251	4303	4245	4245
query52	112	110	105	105
query53	237	259	188	188
query54	492	490	426	426
query55	83	75	78	75
query56	281	288	250	250
query57	1163	1193	1141	1141
query58	265	234	249	234
query59	2806	3049	2732	2732
query60	294	275	255	255
query61	117	111	159	111
query62	733	741	679	679
query63	251	190	190	190
query64	1676	1036	658	658
query65	3275	3143	3112	3112
query66	706	381	292	292
query67	15865	15630	15389	15389
query68	5574	822	532	532
query69	523	315	268	268
query70	1220	1106	1112	1106
query71	427	302	266	266
query72	5426	3614	3777	3614
query73	1284	759	362	362
query74	9258	9136	8759	8759
query75	3459	3165	2673	2673
query76	3799	1159	748	748
query77	540	393	277	277
query78	10007	9897	9267	9267
query79	2552	788	595	595
query80	643	521	440	440
query81	504	272	241	241
query82	360	151	126	126
query83	185	177	156	156
query84	285	87	73	73
query85	777	400	295	295
query86	411	325	275	275
query87	4504	4495	4425	4425
query88	3581	2224	2208	2208
query89	401	313	284	284
query90	1884	197	190	190
query91	129	142	110	110
query92	70	61	58	58
query93	2354	1030	587	587
query94	625	395	298	298
query95	343	268	250	250
query96	480	565	272	272
query97	2785	2881	2761	2761
query98	223	202	200	200
query99	1322	1367	1287	1287
Total cold run time: 296730 ms
Total hot run time: 190340 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7770b3b4cc8fc59e391b4f6cae86f25e2e6c0317, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.09	0.11
query5	0.41	0.43	0.40
query6	1.16	0.67	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.52
query10	0.58	0.58	0.59
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.68	2.80	2.69
query15	0.91	0.86	0.85
query16	0.37	0.39	0.37
query17	1.01	1.02	1.06
query18	0.21	0.19	0.19
query19	1.92	1.81	2.00
query20	0.02	0.01	0.02
query21	15.36	0.90	0.52
query22	0.76	1.15	0.62
query23	15.04	1.38	0.64
query24	7.23	2.89	0.54
query25	0.55	0.11	0.08
query26	0.65	0.17	0.13
query27	0.05	0.04	0.04
query28	10.05	0.76	0.41
query29	12.53	3.95	3.31
query30	0.25	0.09	0.05
query31	2.83	0.58	0.38
query32	3.22	0.55	0.46
query33	3.08	2.99	3.02
query34	16.08	5.16	4.51
query35	4.60	4.54	4.56
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.18 s
Total hot run time: 30.16 s

@zhangstar333
Copy link
Copy Markdown
Contributor Author

zhangstar333 commented Feb 10, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

TPC-H: Total hot run time: 31588 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a2cc5fe35d3a4041f6f9268cd3b30b8e1b07fdb, data reload: false

------ Round 1 ----------------------------------
q1	17578	5233	5016	5016
q2	2047	292	174	174
q3	10429	1222	749	749
q4	10229	996	535	535
q5	7626	2378	2316	2316
q6	191	163	130	130
q7	883	741	597	597
q8	9287	1283	1084	1084
q9	4853	4866	4862	4862
q10	6810	2277	1873	1873
q11	484	289	255	255
q12	342	350	216	216
q13	17776	3707	3070	3070
q14	231	230	211	211
q15	504	477	462	462
q16	630	613	601	601
q17	578	866	335	335
q18	6763	6168	6337	6168
q19	1723	946	547	547
q20	310	322	184	184
q21	2826	2237	1902	1902
q22	361	328	301	301
Total cold run time: 102461 ms
Total hot run time: 31588 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5163	5116	5088	5088
q2	236	321	232	232
q3	2140	2674	2330	2330
q4	1395	1813	1342	1342
q5	4224	4134	4168	4134
q6	208	172	121	121
q7	1857	1804	1643	1643
q8	2618	2738	2566	2566
q9	7201	7077	7142	7077
q10	3023	3200	2808	2808
q11	578	528	511	511
q12	686	826	609	609
q13	3573	3793	3407	3407
q14	288	312	277	277
q15	510	479	478	478
q16	664	682	645	645
q17	1131	1574	1374	1374
q18	7695	7329	7286	7286
q19	797	849	872	849
q20	1966	2023	1870	1870
q21	5476	5003	4859	4859
q22	645	578	524	524
Total cold run time: 52074 ms
Total hot run time: 50030 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

TPC-DS: Total hot run time: 189982 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7a2cc5fe35d3a4041f6f9268cd3b30b8e1b07fdb, data reload: false

query1	1284	946	960	946
query2	6224	1846	1915	1846
query3	11246	4489	4516	4489
query4	55155	24726	22983	22983
query5	4881	565	472	472
query6	316	196	201	196
query7	4883	502	287	287
query8	289	241	217	217
query9	5445	2471	2457	2457
query10	429	326	254	254
query11	15245	15117	15009	15009
query12	159	105	104	104
query13	1032	505	383	383
query14	10178	6599	7007	6599
query15	209	219	198	198
query16	7090	673	499	499
query17	1093	747	598	598
query18	1404	421	320	320
query19	212	205	169	169
query20	131	135	126	126
query21	213	127	116	116
query22	4441	4405	4340	4340
query23	34372	33297	33279	33279
query24	5606	2415	2407	2407
query25	444	441	387	387
query26	679	273	153	153
query27	1856	487	337	337
query28	2835	2429	2399	2399
query29	548	595	445	445
query30	217	189	165	165
query31	891	866	808	808
query32	73	60	61	60
query33	416	359	304	304
query34	777	865	503	503
query35	814	856	735	735
query36	938	1000	912	912
query37	126	107	80	80
query38	4291	4266	4296	4266
query39	1505	1420	1429	1420
query40	203	122	105	105
query41	50	53	50	50
query42	118	109	103	103
query43	508	530	487	487
query44	1329	791	796	791
query45	190	171	170	170
query46	863	1068	646	646
query47	1858	1893	1810	1810
query48	384	415	293	293
query49	706	519	424	424
query50	698	778	419	419
query51	4252	4300	4222	4222
query52	117	104	96	96
query53	247	265	191	191
query54	475	494	410	410
query55	85	79	80	79
query56	253	269	267	267
query57	1161	1191	1131	1131
query58	244	237	240	237
query59	2953	3011	2911	2911
query60	288	275	289	275
query61	148	111	118	111
query62	734	757	677	677
query63	228	191	193	191
query64	1626	1054	704	704
query65	3309	3184	3143	3143
query66	693	397	297	297
query67	15706	15599	15412	15412
query68	4057	780	506	506
query69	479	310	319	310
query70	1163	1119	1104	1104
query71	397	288	253	253
query72	6239	3609	3723	3609
query73	737	749	342	342
query74	8996	8986	8883	8883
query75	3164	3238	2682	2682
query76	2771	1175	724	724
query77	570	368	271	271
query78	10101	10235	9296	9296
query79	1585	873	586	586
query80	861	555	496	496
query81	523	269	242	242
query82	408	159	121	121
query83	182	169	154	154
query84	281	102	76	76
query85	735	353	292	292
query86	372	312	285	285
query87	4499	4468	4419	4419
query88	2775	2193	2158	2158
query89	392	319	286	286
query90	1610	202	187	187
query91	138	138	109	109
query92	64	58	56	56
query93	1261	1005	580	580
query94	642	402	304	304
query95	346	271	259	259
query96	487	559	266	266
query97	2800	2813	2762	2762
query98	237	207	199	199
query99	1311	1396	1245	1245
Total cold run time: 287556 ms
Total hot run time: 189982 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 10, 2025

ClickBench: Total hot run time: 30.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a2cc5fe35d3a4041f6f9268cd3b30b8e1b07fdb, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.42	0.41
query6	1.15	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.61	0.53	0.52
query10	0.57	0.57	0.57
query11	0.15	0.10	0.10
query12	0.14	0.12	0.12
query13	0.62	0.59	0.61
query14	2.81	2.83	2.82
query15	0.93	0.85	0.85
query16	0.38	0.37	0.37
query17	1.05	1.07	0.99
query18	0.22	0.20	0.20
query19	1.86	1.81	1.98
query20	0.01	0.02	0.01
query21	15.41	0.90	0.56
query22	0.76	1.22	0.66
query23	14.90	1.40	0.66
query24	7.65	1.44	0.76
query25	0.48	0.17	0.15
query26	0.66	0.16	0.13
query27	0.05	0.05	0.05
query28	8.95	0.86	0.42
query29	12.56	3.96	3.32
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.46
query33	3.06	2.99	3.11
query34	15.93	5.19	4.54
query35	4.62	4.58	4.58
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.4 s
Total hot run time: 30.81 s

HappenLee
HappenLee previously approved these changes Feb 10, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 10, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Feb 11, 2025
@zhangstar333
Copy link
Copy Markdown
Contributor Author

zhangstar333 commented Feb 11, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 11, 2025

TPC-H: Total hot run time: 31461 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 88fb1d5d998b8f428ee699d30087151ae9df158c, data reload: false

------ Round 1 ----------------------------------
q1	17610	5194	5061	5061
q2	2047	311	171	171
q3	10394	1320	702	702
q4	10215	1003	550	550
q5	7496	2355	2322	2322
q6	185	169	135	135
q7	899	746	594	594
q8	9327	1274	1109	1109
q9	4827	4558	4598	4558
q10	6824	2330	1879	1879
q11	472	267	263	263
q12	357	352	220	220
q13	17760	3680	3040	3040
q14	228	233	214	214
q15	509	476	460	460
q16	619	629	572	572
q17	567	853	355	355
q18	6718	6241	6283	6241
q19	1201	950	537	537
q20	316	318	204	204
q21	2736	2293	1958	1958
q22	364	328	316	316
Total cold run time: 101671 ms
Total hot run time: 31461 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5112	5100	5093	5093
q2	238	328	239	239
q3	2180	2698	2277	2277
q4	1446	1874	1471	1471
q5	4213	4111	4140	4111
q6	202	162	122	122
q7	1834	1831	1674	1674
q8	2589	2593	2473	2473
q9	7334	7019	7053	7019
q10	2997	3201	2749	2749
q11	581	514	489	489
q12	702	775	630	630
q13	3505	3874	3158	3158
q14	282	306	266	266
q15	512	460	472	460
q16	639	667	623	623
q17	1118	1585	1307	1307
q18	7540	7219	7087	7087
q19	801	799	806	799
q20	1957	2024	1840	1840
q21	5314	5092	4802	4802
q22	609	599	569	569
Total cold run time: 51705 ms
Total hot run time: 49258 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 11, 2025

TPC-DS: Total hot run time: 183366 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 88fb1d5d998b8f428ee699d30087151ae9df158c, data reload: false

query1	984	402	367	367
query2	6527	1845	1885	1845
query3	6808	211	211	211
query4	26603	23353	22876	22876
query5	4399	688	475	475
query6	288	192	196	192
query7	4612	502	310	310
query8	301	248	228	228
query9	8610	2512	2512	2512
query10	453	328	262	262
query11	16006	15175	14969	14969
query12	160	123	107	107
query13	1655	546	396	396
query14	9915	6600	6636	6600
query15	220	205	178	178
query16	7676	616	470	470
query17	1193	713	552	552
query18	2002	401	315	315
query19	198	193	168	168
query20	128	121	115	115
query21	206	127	106	106
query22	4319	4320	4239	4239
query23	34022	32967	32886	32886
query24	7696	2395	2409	2395
query25	524	434	399	399
query26	1219	274	158	158
query27	2242	484	329	329
query28	4057	2408	2377	2377
query29	723	557	415	415
query30	237	184	164	164
query31	951	864	783	783
query32	71	70	68	68
query33	580	349	303	303
query34	783	856	519	519
query35	816	809	734	734
query36	955	1033	881	881
query37	122	99	85	85
query38	4031	4230	4141	4141
query39	1467	1387	1401	1387
query40	215	119	108	108
query41	53	57	53	53
query42	120	105	105	105
query43	501	505	486	486
query44	1344	804	790	790
query45	180	166	164	164
query46	867	1028	664	664
query47	1763	1804	1678	1678
query48	389	422	295	295
query49	779	498	420	420
query50	703	745	435	435
query51	4172	4165	4101	4101
query52	102	103	93	93
query53	231	273	188	188
query54	493	515	402	402
query55	82	81	80	80
query56	265	281	264	264
query57	1121	1160	1065	1065
query58	243	249	245	245
query59	2615	2776	2647	2647
query60	286	278	276	276
query61	153	119	113	113
query62	806	737	675	675
query63	235	191	189	189
query64	4278	1032	649	649
query65	3193	3138	3116	3116
query66	1065	413	315	315
query67	15897	15387	15426	15387
query68	7953	777	514	514
query69	471	288	262	262
query70	1217	1134	1109	1109
query71	398	293	273	273
query72	5673	3519	3842	3519
query73	748	753	355	355
query74	9350	9102	8771	8771
query75	3153	3196	2666	2666
query76	3248	1175	759	759
query77	500	385	286	286
query78	9958	10189	9259	9259
query79	2209	844	589	589
query80	610	542	448	448
query81	529	282	241	241
query82	230	152	119	119
query83	188	176	159	159
query84	248	89	73	73
query85	746	347	304	304
query86	372	325	286	286
query87	4438	4606	4371	4371
query88	4022	2223	2257	2223
query89	394	313	295	295
query90	1957	200	196	196
query91	138	143	116	116
query92	82	62	56	56
query93	1798	1016	599	599
query94	680	424	307	307
query95	361	282	257	257
query96	480	562	266	266
query97	2684	2827	2726	2726
query98	234	209	201	201
query99	1291	1395	1241	1241
Total cold run time: 272270 ms
Total hot run time: 183366 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 11, 2025

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 88fb1d5d998b8f428ee699d30087151ae9df158c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.40	0.43	0.40
query6	1.17	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.53	0.50
query10	0.58	0.56	0.56
query11	0.15	0.10	0.10
query12	0.15	0.10	0.11
query13	0.62	0.60	0.60
query14	2.69	2.71	2.69
query15	0.93	0.84	0.86
query16	0.38	0.37	0.38
query17	1.03	1.06	1.03
query18	0.22	0.19	0.20
query19	1.89	1.76	2.01
query20	0.01	0.02	0.01
query21	15.36	0.92	0.54
query22	0.75	1.29	0.98
query23	14.73	1.40	0.61
query24	7.32	1.84	0.50
query25	0.50	0.20	0.12
query26	0.63	0.15	0.14
query27	0.04	0.04	0.04
query28	9.69	0.85	0.42
query29	12.55	4.06	3.41
query30	0.25	0.10	0.07
query31	2.82	0.60	0.38
query32	3.23	0.54	0.46
query33	3.02	3.03	3.06
query34	15.84	5.11	4.55
query35	4.50	4.49	4.46
query36	0.69	0.51	0.49
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.05	0.03	0.02
Total cold run time: 105.22 s
Total hot run time: 30.56 s

HappenLee
HappenLee approved these changes Feb 13, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 13, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 13, 2025

PR approved by at least one committer and no changes requested.

@zhangstar333 zhangstar333 merged commit 16af223 into apache:master Feb 13, 2025
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@zhangstar333 @lzyy2024
[improve](udaf)support class cache for java-udaf (apache#47619)
6772ca3 
### What problem does this PR solve?
Problem Summary:
in some user case,

- the udf-jars package have some resource file, it's maybe about
hundreds MBs,
so if every instances load the jar, it's easy to cause the BE JVM OOM.
- or in some udf-jars have some init operator cause many times, so wants
all instance could only init it's once.

follow up apache#40404
support class cache for java-udaf
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@zhangstar333
[improve](udaf)support class cache for java-udaf (apache#47619)
83c72b8 
### What problem does this PR solve?
Problem Summary:
in some user case,

- the udf-jars package have some resource file, it's maybe about
hundreds MBs,
so if every instances load the jar, it's easy to cause the BE JVM OOM.
- or in some udf-jars have some init operator cause many times, so wants
all instance could only init it's once.

follow up apache#40404
support class cache for java-udaf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLee HappenLee HappenLee approved these changes

@Mryange Mryange Mryange approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zhangstar333 @hello-stephen @doris-robot @HappenLee @Mryange