branch-3.0: [fix](jdbc) make sure init the jdbc client before using it by morningman · Pull Request #51540 · apache/doris

morningman · 2025-06-06T02:34:48Z

This may cause some statistic collection task fail:

2025-06-05 15:02:32,277 WARN (Statistics Job Appender|182) [ExternalCatalog.buildDbForInit():902] Failed to check db DORIS_TEST exist in remote system, ignore it.
java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDatabaseNameList()" because "this.jdbcClient" is null
        at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.listDatabaseNames(JdbcExternalCatalog.java:267) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.ExternalCatalog.getFilteredDatabaseNames(ExternalCatalog.java:476) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.ExternalCatalog.lambda$makeSureInitialized$0(ExternalCatalog.java:318) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?]

Only for branch-3.0.
Fix master in #51471

hello-stephen · 2025-06-06T02:34:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-06-06T02:37:47Z

run buildall

github-actions · 2025-06-06T02:50:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-06T02:50:34Z

PR approved by anyone and no changes requested.

doris-robot · 2025-06-06T03:14:42Z

TPC-H: Total hot run time: 39612 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c6dcd10bdc9ea7613b9af8cfc2313ded6aaeb2c, data reload: false

------ Round 1 ----------------------------------
q1	17574	6695	6534	6534
q2	2060	167	174	167
q3	10755	1094	1112	1094
q4	10543	793	740	740
q5	7751	2864	2876	2864
q6	215	133	132	132
q7	970	621	615	615
q8	9370	1922	2010	1922
q9	6547	6385	6391	6385
q10	6985	2268	2290	2268
q11	462	256	266	256
q12	402	214	214	214
q13	17805	2985	3022	2985
q14	232	199	208	199
q15	513	453	462	453
q16	497	385	380	380
q17	979	518	592	518
q18	7159	6617	6596	6596
q19	1394	1096	1012	1012
q20	469	211	195	195
q21	3987	3331	3128	3128
q22	1078	1004	955	955
Total cold run time: 107747 ms
Total hot run time: 39612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6649	6540	6653	6540
q2	330	241	240	240
q3	3047	2804	2861	2804
q4	2104	1793	1885	1793
q5	5901	5833	5840	5833
q6	218	131	135	131
q7	2235	1802	1821	1802
q8	3370	3546	3513	3513
q9	9035	8803	8913	8803
q10	3547	3501	3513	3501
q11	601	510	482	482
q12	788	590	610	590
q13	10665	3218	3179	3179
q14	307	264	274	264
q15	520	469	477	469
q16	509	453	435	435
q17	1832	1624	1618	1618
q18	8254	7787	7662	7662
q19	1692	1471	1570	1471
q20	2150	1821	1799	1799
q21	5306	5401	5309	5309
q22	1146	1022	1040	1022
Total cold run time: 70206 ms
Total hot run time: 59260 ms

doris-robot · 2025-06-06T03:27:16Z

TPC-DS: Total hot run time: 196323 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c6dcd10bdc9ea7613b9af8cfc2313ded6aaeb2c, data reload: false

query1	1276	898	917	898
query2	6344	1958	1873	1873
query3	10790	4332	4223	4223
query4	60979	28970	23528	23528
query5	5281	475	457	457
query6	405	198	196	196
query7	5430	334	319	319
query8	309	227	235	227
query9	8284	2604	2583	2583
query10	455	299	260	260
query11	18120	15165	15717	15165
query12	172	103	106	103
query13	1461	461	468	461
query14	10403	6619	7387	6619
query15	197	181	173	173
query16	7186	500	490	490
query17	1118	574	576	574
query18	1893	309	313	309
query19	211	158	151	151
query20	114	112	108	108
query21	202	101	101	101
query22	4805	4447	4556	4447
query23	35311	33797	34298	33797
query24	6146	2909	2919	2909
query25	556	442	413	413
query26	656	169	168	168
query27	1792	368	358	358
query28	4083	2234	2182	2182
query29	697	472	457	457
query30	244	155	164	155
query31	994	811	836	811
query32	71	58	59	58
query33	487	288	305	288
query34	912	529	538	529
query35	862	745	727	727
query36	1078	962	975	962
query37	109	70	73	70
query38	4069	3948	3932	3932
query39	1516	1470	1465	1465
query40	210	110	103	103
query41	52	51	56	51
query42	120	105	113	105
query43	532	491	506	491
query44	1185	820	812	812
query45	185	166	176	166
query46	1145	737	731	731
query47	2041	1928	1964	1928
query48	468	387	384	384
query49	716	401	385	385
query50	826	415	438	415
query51	7491	7306	7141	7141
query52	105	90	93	90
query53	271	189	189	189
query54	578	458	480	458
query55	83	81	82	81
query56	285	283	259	259
query57	1254	1144	1146	1144
query58	223	203	218	203
query59	3205	3057	3122	3057
query60	282	253	261	253
query61	132	112	110	110
query62	782	695	693	693
query63	233	251	202	202
query64	1416	682	647	647
query65	3259	3187	3173	3173
query66	659	302	301	301
query67	15822	15827	15673	15673
query68	4229	606	586	586
query69	415	272	275	272
query70	1161	1097	1127	1097
query71	336	276	259	259
query72	6107	4073	4013	4013
query73	750	348	360	348
query74	10351	9131	9245	9131
query75	3273	2655	2674	2655
query76	2040	1052	1030	1030
query77	507	302	278	278
query78	10646	9547	9559	9547
query79	2075	614	597	597
query80	1307	424	461	424
query81	511	232	221	221
query82	578	90	87	87
query83	158	148	139	139
query84	289	81	73	73
query85	970	309	294	294
query86	352	288	297	288
query87	4406	4220	4273	4220
query88	4001	2365	2348	2348
query89	425	300	288	288
query90	1985	187	186	186
query91	181	151	151	151
query92	67	47	50	47
query93	2603	561	551	551
query94	716	279	311	279
query95	353	256	259	256
query96	617	276	277	276
query97	3329	3121	3169	3121
query98	208	200	197	197
query99	1573	1273	1310	1273
Total cold run time: 315097 ms
Total hot run time: 196323 ms

doris-robot · 2025-06-06T03:32:37Z

ClickBench: Total hot run time: 30.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c6dcd10bdc9ea7613b9af8cfc2313ded6aaeb2c, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.52	0.52	0.51
query6	1.15	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.55	0.57
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	0.80	0.80	0.81
query15	0.84	0.83	0.84
query16	0.38	0.39	0.40
query17	1.01	1.02	1.02
query18	0.24	0.23	0.22
query19	1.82	1.73	1.86
query20	0.01	0.01	0.01
query21	15.46	0.61	0.60
query22	2.69	2.64	1.71
query23	17.02	0.98	0.79
query24	3.08	1.97	1.22
query25	0.35	0.08	0.12
query26	0.49	0.13	0.16
query27	0.05	0.04	0.05
query28	9.32	0.48	0.50
query29	12.91	3.22	3.22
query30	0.24	0.07	0.06
query31	2.87	0.37	0.39
query32	3.27	0.46	0.46
query33	2.98	3.00	2.99
query34	17.14	4.44	4.54
query35	4.48	4.52	4.48
query36	0.66	0.50	0.48
query37	0.09	0.05	0.05
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.29 s
Total hot run time: 30.2 s

dataroaring · 2025-06-11T03:04:19Z

run buildall

doris-robot · 2025-06-11T04:22:12Z

TPC-H: Total hot run time: 39936 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52dd35010851456e212baed02ee99a16c485cd5e, data reload: false

------ Round 1 ----------------------------------
q1	17583	7053	6627	6627
q2	2074	169	152	152
q3	10624	1091	1207	1091
q4	10554	766	731	731
q5	7778	2923	2901	2901
q6	218	135	129	129
q7	998	645	617	617
q8	9364	1956	2038	1956
q9	6651	6397	6399	6397
q10	7041	2279	2270	2270
q11	459	251	258	251
q12	392	205	206	205
q13	17788	2952	2999	2952
q14	230	209	220	209
q15	499	468	464	464
q16	483	386	374	374
q17	992	569	547	547
q18	7333	6609	6640	6609
q19	1485	1138	1037	1037
q20	457	212	212	212
q21	4002	3224	3247	3224
q22	1067	981	983	981
Total cold run time: 108072 ms
Total hot run time: 39936 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6598	6581	6571	6571
q2	326	224	229	224
q3	2967	2775	2966	2775
q4	2016	1839	1787	1787
q5	5785	5750	5767	5750
q6	211	129	128	128
q7	2248	1782	1837	1782
q8	3382	3561	3525	3525
q9	8944	8795	8916	8795
q10	3535	3528	3493	3493
q11	589	476	493	476
q12	809	617	582	582
q13	10248	3117	3175	3117
q14	299	280	268	268
q15	511	468	462	462
q16	495	442	434	434
q17	1855	1641	1624	1624
q18	8234	7830	7649	7649
q19	1694	1532	1594	1532
q20	2051	1809	1824	1809
q21	5556	5319	5289	5289
q22	1127	1018	1019	1018
Total cold run time: 69480 ms
Total hot run time: 59090 ms

doris-robot · 2025-06-11T04:34:47Z

TPC-DS: Total hot run time: 196691 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52dd35010851456e212baed02ee99a16c485cd5e, data reload: false

query1	1309	906	890	890
query2	6339	1965	1931	1931
query3	10815	4228	4247	4228
query4	60688	29559	23381	23381
query5	5212	442	449	442
query6	394	165	175	165
query7	5507	308	304	304
query8	306	219	225	219
query9	8562	2603	2584	2584
query10	447	267	253	253
query11	17630	15488	15676	15488
query12	162	108	107	107
query13	1418	451	425	425
query14	10439	7415	7255	7255
query15	198	181	175	175
query16	7107	481	483	481
query17	1173	577	576	576
query18	1819	340	323	323
query19	207	164	159	159
query20	123	117	109	109
query21	214	108	106	106
query22	4717	4383	4817	4383
query23	34678	33875	33983	33875
query24	6201	2902	2920	2902
query25	544	421	425	421
query26	668	170	168	168
query27	2108	357	359	357
query28	4251	2211	2197	2197
query29	707	474	447	447
query30	239	157	154	154
query31	978	786	833	786
query32	66	59	56	56
query33	426	283	288	283
query34	928	516	533	516
query35	855	718	721	718
query36	1091	927	928	927
query37	112	68	67	67
query38	4099	3959	3956	3956
query39	1521	1454	1484	1454
query40	206	100	102	100
query41	50	51	47	47
query42	113	107	103	103
query43	518	476	487	476
query44	1191	831	827	827
query45	189	174	170	170
query46	1171	767	704	704
query47	2040	1907	1931	1907
query48	474	377	397	377
query49	735	393	402	393
query50	830	418	424	418
query51	7577	7302	7217	7217
query52	100	93	93	93
query53	258	190	184	184
query54	583	459	471	459
query55	75	77	82	77
query56	277	245	264	245
query57	1275	1192	1124	1124
query58	227	204	216	204
query59	3266	3122	2977	2977
query60	283	258	261	258
query61	108	105	108	105
query62	804	686	662	662
query63	215	185	185	185
query64	1413	655	624	624
query65	3275	3171	3154	3154
query66	710	314	295	295
query67	15977	15718	15484	15484
query68	4073	586	584	584
query69	438	258	261	258
query70	1168	1070	1150	1070
query71	356	261	259	259
query72	6344	4295	4066	4066
query73	753	351	354	351
query74	10170	9106	9355	9106
query75	3376	2647	2645	2645
query76	1982	1070	977	977
query77	457	265	267	265
query78	10510	9622	9633	9622
query79	2144	602	607	602
query80	1360	441	426	426
query81	504	219	215	215
query82	1250	83	88	83
query83	270	142	139	139
query84	278	75	76	75
query85	1002	299	288	288
query86	371	296	304	296
query87	4431	4253	4241	4241
query88	3950	2384	2382	2382
query89	419	299	290	290
query90	1996	185	188	185
query91	182	148	148	148
query92	63	47	49	47
query93	2985	560	553	553
query94	773	283	300	283
query95	355	260	258	258
query96	621	281	273	273
query97	3287	3112	3132	3112
query98	214	191	198	191
query99	1585	1296	1264	1264
Total cold run time: 315865 ms
Total hot run time: 196691 ms

doris-robot · 2025-06-11T04:40:07Z

ClickBench: Total hot run time: 30.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52dd35010851456e212baed02ee99a16c485cd5e, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.53	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.04
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.54	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.59
query14	0.78	0.77	0.78
query15	0.84	0.84	0.82
query16	0.40	0.38	0.38
query17	1.00	1.06	1.01
query18	0.23	0.20	0.21
query19	1.89	1.89	1.75
query20	0.01	0.01	0.01
query21	15.40	0.59	0.62
query22	3.02	2.13	1.92
query23	16.94	0.92	0.88
query24	3.09	0.96	1.47
query25	0.21	0.24	0.07
query26	0.58	0.13	0.13
query27	0.04	0.05	0.04
query28	10.00	0.55	0.44
query29	12.64	3.19	3.14
query30	0.25	0.06	0.05
query31	2.86	0.39	0.37
query32	3.25	0.46	0.48
query33	3.02	2.96	3.01
query34	17.10	4.44	4.48
query35	4.51	4.54	4.52
query36	0.64	0.51	0.49
query37	0.08	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.81 s
Total hot run time: 30.05 s

morningman · 2025-06-19T07:25:19Z

run buildall

dataroaring

LGTM

morningman requested a review from dataroaring as a code owner June 6, 2025 02:34

morningman added dev/3.0.x p0_test labels Jun 6, 2025

morningman changed the title ~~branch-3.0: make sure init the jdbc client before using it~~ branch-3.0: [fix](jdbc) make sure init the jdbc client before using it Jun 6, 2025

zy-kkk previously approved these changes Jun 6, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jun 6, 2025

github-actions Bot added the reviewed label Jun 6, 2025

dataroaring force-pushed the 30_jdbc_init branch from 0c6dcd1 to 52dd350 Compare June 11, 2025 03:04

morningman marked this pull request as draft June 17, 2025 08:34

morningman added 2 commits June 19, 2025 15:24

branch-3.0: make sure init the jdbc client before using it

de9e22b

1

e8ce553

morningman dismissed zy-kkk’s stale review via e8ce553 June 19, 2025 07:24

morningman force-pushed the 30_jdbc_init branch from 52dd350 to e8ce553 Compare June 19, 2025 07:24

morningman marked this pull request as ready for review June 19, 2025 07:25

github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Jun 19, 2025

dataroaring approved these changes Jun 19, 2025

View reviewed changes

dataroaring merged commit f440b74 into apache:branch-3.0 Jun 19, 2025
21 of 25 checks passed

morningman removed the dev/3.0.x label Jun 24, 2025

morningman added the dev/3.0.7-merged label Jun 24, 2025

morrySnow removed the dev/3.0.7-merged label Jun 24, 2025

Conversation

morningman commented Jun 6, 2025

Uh oh!

hello-stephen commented Jun 6, 2025

Uh oh!

morningman commented Jun 6, 2025

Uh oh!

github-actions Bot commented Jun 6, 2025

Uh oh!

github-actions Bot commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

doris-robot commented Jun 6, 2025

Uh oh!

dataroaring commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

morningman commented Jun 19, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants