[fix](catalog) close connection on refresh by morningman · Pull Request #35426 · apache/doris

morningman · 2024-05-27T05:05:29Z

Proposed changes

When refresh the catalog, the related client(such as HMSClient) should be closed,
or the connection may be leaked.
Remove some unused code related to deprecated hive external table

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-27T05:05:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman · 2024-05-27T05:19:55Z

run buildall

doris-robot · 2024-05-27T05:58:27Z

TPC-H: Total hot run time: 41591 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff762da79273ae5cd35cec221b590b5e4c4ed8c9, data reload: false

------ Round 1 ----------------------------------
q1	17606	4511	4233	4233
q2	2021	188	192	188
q3	10437	1254	1164	1164
q4	10195	821	791	791
q5	7480	2752	2684	2684
q6	229	130	131	130
q7	938	607	622	607
q8	9229	2106	2065	2065
q9	9267	6632	6652	6632
q10	9029	3946	3855	3855
q11	466	238	243	238
q12	438	237	232	232
q13	17272	3268	3179	3179
q14	280	224	246	224
q15	522	483	489	483
q16	518	398	408	398
q17	975	708	608	608
q18	8224	7760	7764	7760
q19	3667	1537	1508	1508
q20	624	317	323	317
q21	5055	4016	4046	4016
q22	353	291	279	279
Total cold run time: 114825 ms
Total hot run time: 41591 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4505	4348	4404	4348
q2	374	291	277	277
q3	3092	2927	2889	2889
q4	2014	1724	1642	1642
q5	5291	5475	5505	5475
q6	213	127	131	127
q7	2214	1852	1883	1852
q8	3196	3433	3411	3411
q9	8641	8651	8649	8649
q10	4094	3880	3746	3746
q11	584	486	492	486
q12	778	590	616	590
q13	16820	3169	3191	3169
q14	303	271	279	271
q15	514	493	489	489
q16	528	434	428	428
q17	1814	1555	1537	1537
q18	7841	7674	7458	7458
q19	1682	1565	1529	1529
q20	1996	1762	1786	1762
q21	10364	4853	4707	4707
q22	584	494	504	494
Total cold run time: 77442 ms
Total hot run time: 55336 ms

doris-robot · 2024-05-27T06:08:59Z

TPC-DS: Total hot run time: 169193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff762da79273ae5cd35cec221b590b5e4c4ed8c9, data reload: false

query1	929	396	387	387
query2	6438	2500	2360	2360
query3	6631	202	208	202
query4	20616	17279	17233	17233
query5	4115	439	438	438
query6	248	157	162	157
query7	4582	296	303	296
query8	243	199	197	197
query9	8500	2359	2340	2340
query10	446	281	260	260
query11	10530	10059	9964	9964
query12	137	92	89	89
query13	1647	375	381	375
query14	9369	6473	7804	6473
query15	230	171	172	171
query16	7748	269	269	269
query17	1539	550	533	533
query18	1972	292	314	292
query19	199	161	154	154
query20	97	87	87	87
query21	194	138	129	129
query22	4042	3926	3883	3883
query23	33392	33191	33137	33137
query24	6694	2858	2931	2858
query25	472	357	361	357
query26	694	162	160	160
query27	1904	331	329	329
query28	3770	2084	2040	2040
query29	850	616	601	601
query30	215	150	152	150
query31	994	770	740	740
query32	59	53	55	53
query33	499	273	273	273
query34	880	473	495	473
query35	729	602	628	602
query36	1024	936	943	936
query37	111	65	67	65
query38	2913	2827	2722	2722
query39	866	790	815	790
query40	201	127	133	127
query41	47	43	48	43
query42	105	103	95	95
query43	569	563	574	563
query44	1120	742	769	742
query45	184	166	159	159
query46	1051	726	714	714
query47	1827	1767	1770	1767
query48	376	311	305	305
query49	772	400	399	399
query50	780	393	404	393
query51	6865	6757	6830	6757
query52	99	94	92	92
query53	357	288	296	288
query54	533	436	427	427
query55	77	73	74	73
query56	271	254	251	251
query57	1105	1042	1032	1032
query58	242	225	234	225
query59	3793	3086	3235	3086
query60	275	258	260	258
query61	94	136	88	88
query62	564	475	465	465
query63	317	294	296	294
query64	2587	1757	1711	1711
query65	3192	3060	3133	3060
query66	785	330	323	323
query67	15151	14771	14760	14760
query68	4630	529	537	529
query69	469	276	279	276
query70	1148	1101	1154	1101
query71	393	275	277	275
query72	7651	5883	2717	2717
query73	731	324	321	321
query74	6078	5525	5612	5525
query75	3391	2673	2666	2666
query76	2286	1131	1002	1002
query77	435	268	277	268
query78	10209	9740	10277	9740
query79	2211	519	513	513
query80	926	459	439	439
query81	514	235	230	230
query82	769	91	93	91
query83	252	173	171	171
query84	251	89	86	86
query85	1037	273	275	273
query86	472	308	336	308
query87	3276	3101	3085	3085
query88	3952	2446	2445	2445
query89	469	400	397	397
query90	1990	199	194	194
query91	127	98	99	98
query92	63	52	50	50
query93	1472	497	493	493
query94	1117	191	189	189
query95	410	311	310	310
query96	598	273	272	272
query97	3189	2991	2987	2987
query98	234	244	217	217
query99	1205	853	859	853
Total cold run time: 251812 ms
Total hot run time: 169193 ms

doris-robot · 2024-05-27T06:14:28Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff762da79273ae5cd35cec221b590b5e4c4ed8c9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.07
query5	0.50	0.50	0.50
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.53	0.55	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.78	0.78	0.79
query15	0.83	0.81	0.80
query16	0.36	0.35	0.34
query17	0.96	0.98	0.96
query18	0.24	0.26	0.22
query19	1.77	1.67	1.67
query20	0.01	0.01	0.01
query21	15.44	0.70	0.67
query22	4.73	7.17	2.06
query23	18.31	1.36	1.26
query24	1.36	0.40	0.23
query25	0.16	0.09	0.08
query26	0.26	0.17	0.17
query27	0.09	0.08	0.09
query28	13.35	1.02	0.99
query29	12.75	3.31	3.30
query30	0.24	0.05	0.06
query31	2.88	0.40	0.38
query32	3.27	0.48	0.47
query33	2.90	2.84	2.91
query34	17.16	4.40	4.47
query35	4.50	4.48	4.58
query36	0.68	0.50	0.46
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.15
query41	0.09	0.04	0.04
query42	0.07	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.45 s
Total hot run time: 30.62 s

zddr

Can't we keep it for use next time we refresh the data？

Jibing-Li

LGTM

github-actions · 2024-05-28T02:38:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-28T02:38:34Z

PR approved by anyone and no changes requested.

morningman · 2024-05-28T06:16:52Z

Can't we keep it for use next time we refresh the data？

Sounds reasonable, I will try refactor it.

morningman · 2024-05-30T07:55:25Z

run buildall

doris-robot · 2024-05-30T10:29:00Z

TPC-H: Total hot run time: 41996 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26914fd475c2ba53789d98a340d6292d2a37839e, data reload: false

------ Round 1 ----------------------------------
q1	17604	4255	4249	4249
q2	2020	193	194	193
q3	10452	1264	1137	1137
q4	10190	822	809	809
q5	7480	2692	2703	2692
q6	228	132	141	132
q7	949	611	616	611
q8	9219	2134	2099	2099
q9	9975	6739	6708	6708
q10	9393	3900	3823	3823
q11	448	244	246	244
q12	460	225	239	225
q13	17529	3219	3219	3219
q14	272	206	221	206
q15	504	470	475	470
q16	488	385	403	385
q17	981	699	742	699
q18	8325	7938	7819	7819
q19	6412	1598	1608	1598
q20	668	331	322	322
q21	5079	4021	4124	4021
q22	394	349	335	335
Total cold run time: 119070 ms
Total hot run time: 41996 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4603	4437	4429	4429
q2	374	289	281	281
q3	3211	2857	2802	2802
q4	1918	1626	1660	1626
q5	5486	5486	5497	5486
q6	210	131	136	131
q7	2216	1787	1817	1787
q8	3244	3415	3394	3394
q9	8647	8685	8618	8618
q10	3912	3814	3873	3814
q11	592	497	504	497
q12	775	635	619	619
q13	17118	3172	3241	3172
q14	298	294	273	273
q15	515	479	494	479
q16	475	416	411	411
q17	1790	1488	1470	1470
q18	7724	7552	7567	7552
q19	1682	1528	1570	1528
q20	1977	1781	1808	1781
q21	8363	4812	4776	4776
q22	610	536	519	519
Total cold run time: 75740 ms
Total hot run time: 55445 ms

doris-robot · 2024-05-30T10:39:34Z

TPC-DS: Total hot run time: 172903 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26914fd475c2ba53789d98a340d6292d2a37839e, data reload: false

query1	924	375	367	367
query2	6437	2329	2300	2300
query3	6643	216	214	214
query4	19752	17224	17293	17224
query5	4203	432	432	432
query6	277	170	169	169
query7	4575	299	287	287
query8	302	284	274	274
query9	8510	2415	2393	2393
query10	461	283	257	257
query11	10749	10100	10041	10041
query12	140	89	85	85
query13	1642	356	361	356
query14	10163	7537	7653	7537
query15	232	194	185	185
query16	7289	273	257	257
query17	1292	524	502	502
query18	1890	276	269	269
query19	199	164	148	148
query20	91	83	94	83
query21	204	132	128	128
query22	4389	4231	4118	4118
query23	33675	33134	33163	33134
query24	5068	2884	2950	2884
query25	470	383	412	383
query26	701	151	155	151
query27	1845	317	317	317
query28	3742	2087	2080	2080
query29	820	605	598	598
query30	235	151	151	151
query31	931	758	769	758
query32	62	53	56	53
query33	431	270	268	268
query34	836	480	474	474
query35	688	613	620	613
query36	1019	917	911	911
query37	103	65	66	65
query38	2926	2813	2802	2802
query39	841	810	796	796
query40	197	125	122	122
query41	52	48	50	48
query42	101	96	97	96
query43	556	543	524	524
query44	1065	741	759	741
query45	179	171	160	160
query46	1064	712	712	712
query47	1900	1764	1845	1764
query48	375	300	291	291
query49	751	397	384	384
query50	780	383	389	383
query51	6721	6671	6609	6609
query52	105	86	100	86
query53	355	278	300	278
query54	537	438	438	438
query55	85	76	76	76
query56	258	242	238	238
query57	1153	1026	1041	1026
query58	252	205	204	204
query59	3350	3119	3088	3088
query60	276	261	259	259
query61	91	90	86	86
query62	528	454	448	448
query63	327	289	297	289
query64	2547	1750	1776	1750
query65	3179	3103	3114	3103
query66	702	331	337	331
query67	15271	14948	14880	14880
query68	4538	545	542	542
query69	438	282	270	270
query70	1168	1030	1089	1030
query71	392	267	286	267
query72	7882	5688	5156	5156
query73	709	323	323	323
query74	5954	5642	5617	5617
query75	3287	2596	2644	2596
query76	2274	979	1034	979
query77	387	268	274	268
query78	10064	9980	9746	9746
query79	2382	521	518	518
query80	1172	515	433	433
query81	516	223	217	217
query82	812	92	91	91
query83	248	173	173	173
query84	245	87	82	82
query85	973	269	267	267
query86	445	332	327	327
query87	3320	3118	3113	3113
query88	4189	2354	2356	2354
query89	488	392	394	392
query90	2066	186	190	186
query91	124	97	96	96
query92	63	49	48	48
query93	2091	514	504	504
query94	1176	193	185	185
query95	458	311	305	305
query96	597	269	265	265
query97	3159	3000	3011	3000
query98	237	219	217	217
query99	1210	873	858	858
Total cold run time: 250578 ms
Total hot run time: 172903 ms

doris-robot · 2024-05-30T10:45:05Z

ClickBench: Total hot run time: 31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26914fd475c2ba53789d98a340d6292d2a37839e, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.54	0.50	0.51
query6	1.12	0.72	0.72
query7	0.01	0.02	0.01
query8	0.04	0.04	0.04
query9	0.53	0.50	0.49
query10	0.54	0.56	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.76
query15	0.83	0.80	0.80
query16	0.35	0.36	0.37
query17	1.03	0.95	1.02
query18	0.21	0.25	0.22
query19	1.83	1.74	1.76
query20	0.02	0.01	0.01
query21	15.44	0.71	0.68
query22	4.24	6.84	2.35
query23	18.33	1.33	1.34
query24	1.62	0.21	0.29
query25	0.12	0.09	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.09
query28	13.38	1.01	1.01
query29	13.16	3.31	3.21
query30	0.24	0.06	0.05
query31	2.85	0.40	0.38
query32	3.27	0.47	0.46
query33	2.87	2.87	2.90
query34	17.25	4.45	4.45
query35	4.52	4.49	4.64
query36	0.67	0.46	0.46
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.16	0.16	0.15
query41	0.09	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.03	0.03
Total cold run time: 109.76 s
Total hot run time: 31 s

github-actions · 2024-05-30T10:52:01Z

PR approved by at least one committer and no changes requested.

1. When refresh the catalog, the related client(such as HMSClient) should be closed, or the connection may be leaked. 2. Remove some unused code related to deprecated hive external table

bp #35426

morningman force-pushed the refresh_close branch from 41b12be to ff762da Compare May 27, 2024 05:17

morningman marked this pull request as ready for review May 27, 2024 05:19

zddr reviewed May 27, 2024

View reviewed changes

Jibing-Li previously approved these changes May 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2024

github-actions bot added the reviewed label May 28, 2024

morningman added the dev/2.1.x label May 28, 2024

zddr approved these changes May 30, 2024

View reviewed changes

[fix](catalog) close connection on refresh

26914fd

morningman dismissed Jibing-Li’s stale review via 26914fd May 30, 2024 07:55

morningman force-pushed the refresh_close branch from ff762da to 26914fd Compare May 30, 2024 07:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 30, 2024

Jibing-Li approved these changes May 30, 2024

View reviewed changes

zddr approved these changes May 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 30, 2024

morningman merged commit 754b55f into apache:master May 30, 2024

yiguolei added the 2.1.0-conflict label May 30, 2024

morningman mentioned this pull request May 31, 2024

[fix](catalog) close connection on refresh (#35426) #35743

Merged

yiguolei pushed a commit that referenced this pull request May 31, 2024

[fix](catalog) close connection on refresh (#35426) (#35743)

1d89dd7

bp #35426

morningman added dev/2.1.4-merged and removed dev/2.1.x 2.1.0-conflict labels May 31, 2024

morningman mentioned this pull request Jun 1, 2024

Release Notes 2.1.4 #35752

Closed

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Conversation

morningman commented May 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

morningman commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

zddr left a comment

Choose a reason for hiding this comment

Uh oh!

Jibing-Li left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 28, 2024

Uh oh!

github-actions bot commented May 28, 2024

Uh oh!

morningman commented May 28, 2024

Uh oh!

morningman commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

github-actions bot commented May 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

morningman commented May 27, 2024 •

edited

Loading