Skip to content

[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable.#41041

Merged
yiguolei merged 1 commit intoapache:masterfrom
kaka11chen:fix_jdbc_connector_memory_leak
Sep 21, 2024
Merged

[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable.#41041
yiguolei merged 1 commit intoapache:masterfrom
kaka11chen:fix_jdbc_connector_memory_leak

Conversation

@kaka11chen
Copy link
Copy Markdown
Contributor

@kaka11chen kaka11chen commented Sep 20, 2024

Proposed changes

[Fix] (jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaka11chen
Copy link
Copy Markdown
Contributor Author

kaka11chen commented Sep 20, 2024

run buildall

@kaka11chen kaka11chen changed the title [Fix] (jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable. [Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable. Sep 20, 2024
zy-kkk
zy-kkk approved these changes Sep 20, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@zy-kkk zy-kkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 20, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 20, 2024

PR approved by anyone and no changes requested.

morningman
morningman approved these changes Sep 20, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 20, 2024

TPC-H: Total hot run time: 41907 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97dc45e8b00008409a06b81e8f4b3a802325bd2b, data reload: false

------ Round 1 ----------------------------------
q1	17747	7460	7306	7306
q2	2068	167	166	166
q3	10770	1112	1279	1112
q4	10381	775	746	746
q5	7756	3111	3042	3042
q6	239	153	152	152
q7	1019	630	614	614
q8	9433	2075	2144	2075
q9	6863	6403	6417	6403
q10	7019	2321	2292	2292
q11	432	249	263	249
q12	412	220	226	220
q13	17776	3034	2993	2993
q14	252	219	223	219
q15	582	539	513	513
q16	669	611	638	611
q17	996	796	789	789
q18	7485	6759	6947	6759
q19	1400	1118	1024	1024
q20	603	305	289	289
q21	4201	3342	3371	3342
q22	1109	996	991	991
Total cold run time: 109212 ms
Total hot run time: 41907 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7274	7311	7274
q2	334	237	246	237
q3	3048	3021	2938	2938
q4	2074	1855	1861	1855
q5	5617	5600	5670	5600
q6	233	147	145	145
q7	2164	1773	1808	1773
q8	3313	3429	3419	3419
q9	8687	8982	8792	8792
q10	3475	3507	3516	3507
q11	584	492	492	492
q12	820	626	637	626
q13	10837	3174	3139	3139
q14	301	291	264	264
q15	580	521	505	505
q16	713	685	686	685
q17	1822	1584	1590	1584
q18	8357	7816	7752	7752
q19	1759	1539	1586	1539
q20	2115	1901	1930	1901
q21	5502	5654	5523	5523
q22	1122	1043	1009	1009
Total cold run time: 70726 ms
Total hot run time: 60559 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 20, 2024

TPC-DS: Total hot run time: 194575 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97dc45e8b00008409a06b81e8f4b3a802325bd2b, data reload: false

query1	1279	890	893	890
query2	6339	1987	1986	1986
query3	10750	3779	3985	3779
query4	63407	26972	23569	23569
query5	5237	470	480	470
query6	404	165	164	164
query7	5477	304	292	292
query8	308	243	225	225
query9	9068	2644	2644	2644
query10	459	296	272	272
query11	17824	15152	15840	15152
query12	156	101	113	101
query13	1463	444	423	423
query14	10829	6740	7776	6740
query15	212	182	183	182
query16	6709	490	487	487
query17	1114	588	595	588
query18	1474	314	315	314
query19	215	159	147	147
query20	118	113	123	113
query21	210	111	102	102
query22	4980	4569	4913	4569
query23	34738	33857	33991	33857
query24	5986	2958	2800	2800
query25	516	414	397	397
query26	633	159	160	159
query27	1609	285	288	285
query28	4127	2484	2445	2445
query29	692	439	415	415
query30	229	153	161	153
query31	972	762	802	762
query32	81	52	56	52
query33	395	290	284	284
query34	908	489	480	480
query35	872	726	719	719
query36	1057	945	941	941
query37	152	87	86	86
query38	4111	3907	3902	3902
query39	1484	1408	1399	1399
query40	210	100	96	96
query41	49	49	49	49
query42	122	97	93	93
query43	526	485	480	480
query44	1177	835	811	811
query45	190	163	162	162
query46	1138	762	758	758
query47	1931	1837	1827	1827
query48	469	357	362	357
query49	705	403	408	403
query50	849	411	408	408
query51	7124	7026	6994	6994
query52	97	88	82	82
query53	247	177	178	177
query54	560	447	449	447
query55	73	75	74	74
query56	260	257	245	245
query57	1222	1113	1105	1105
query58	213	257	247	247
query59	3151	2928	2811	2811
query60	290	268	252	252
query61	103	107	107	107
query62	753	647	638	638
query63	225	186	180	180
query64	1395	644	643	643
query65	3252	3198	3167	3167
query66	679	302	301	301
query67	15741	15528	15458	15458
query68	4051	603	555	555
query69	601	357	304	304
query70	1141	1120	1076	1076
query71	413	273	278	273
query72	7019	4024	4083	4024
query73	786	332	321	321
query74	9818	9049	8991	8991
query75	3471	2721	2671	2671
query76	2484	854	991	854
query77	623	290	293	290
query78	9973	9208	9190	9190
query79	1248	545	547	545
query80	862	448	438	438
query81	526	245	244	244
query82	1318	139	144	139
query83	329	135	139	135
query84	283	82	73	73
query85	878	297	287	287
query86	334	297	298	297
query87	4570	4340	4299	4299
query88	3398	2312	2305	2305
query89	401	290	285	285
query90	1916	190	183	183
query91	198	144	142	142
query92	61	48	48	48
query93	1602	541	533	533
query94	742	256	299	256
query95	337	250	259	250
query96	623	278	271	271
query97	3265	3098	3168	3098
query98	216	196	196	196
query99	1546	1285	1293	1285
Total cold run time: 314378 ms
Total hot run time: 194575 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 20, 2024

ClickBench: Total hot run time: 33.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97dc45e8b00008409a06b81e8f4b3a802325bd2b, data reload: false

query1	0.05	0.04	0.04
query2	0.05	0.03	0.03
query3	0.23	0.06	0.07
query4	1.65	0.10	0.11
query5	0.50	0.49	0.50
query6	1.13	0.72	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.58	0.54
query11	0.14	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.59	0.58
query14	3.09	3.00	3.02
query15	0.90	0.83	0.81
query16	0.40	0.37	0.38
query17	1.03	1.08	1.01
query18	0.19	0.19	0.20
query19	1.81	1.79	1.99
query20	0.01	0.01	0.01
query21	15.70	0.57	0.56
query22	2.80	2.12	2.59
query23	17.70	0.82	0.71
query24	3.32	1.69	1.19
query25	0.42	0.18	0.11
query26	0.28	0.14	0.12
query27	0.04	0.04	0.04
query28	9.88	1.09	1.08
query29	12.61	3.23	3.21
query30	0.25	0.06	0.05
query31	2.89	0.38	0.39
query32	3.26	0.46	0.46
query33	3.00	3.01	3.03
query34	17.09	4.34	4.44
query35	4.45	4.40	4.39
query36	0.67	0.51	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.93 s
Total hot run time: 33.06 s

@kaka11chen
Copy link
Copy Markdown
Contributor Author

kaka11chen commented Sep 20, 2024

run external

1 similar comment
@suxiaogang223
Copy link
Copy Markdown
Contributor

suxiaogang223 commented Sep 20, 2024

run external

@yiguolei yiguolei merged commit 6bceb32 into apache:master Sep 21, 2024
@wm1581066 wm1581066 added the p0_c label Sep 22, 2024
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Sep 25, 2024
@kaka11chen
[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't cl…
bf08de3 
…ose `outputTable`. (apache#41041)

## Proposed changes

[Fix] (jdbc-scanner) Fix jdbc scanner memory leak because it didn't
close `outputTable`.
@kaka11chen kaka11chen mentioned this pull request Sep 25, 2024
Merged
hello-stephen pushed a commit that referenced this pull request Sep 25, 2024
@kaka11chen
[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't cl…
306af52 
…ose `outputTable`. (#41266)

## Proposed changes

Backport #41041.
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@kaka11chen @dataroaring
[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't cl…
365ada1 
…ose `outputTable`. (#41041)

## Proposed changes

[Fix] (jdbc-scanner) Fix jdbc scanner memory leak because it didn't
close `outputTable`.
zy-kkk pushed a commit to zy-kkk/doris that referenced this pull request Oct 8, 2024
@kaka11chen @zy-kkk
[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't cl…
aca6142 
…ose `outputTable`. (apache#41266)

## Proposed changes

Backport apache#41041.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
morningman pushed a commit to morningman/doris that referenced this pull request Nov 21, 2024
@kaka11chen @morningman
[Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't cl…
86472ba 
…ose `outputTable`. (apache#41266)

## Proposed changes

Backport apache#41041.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.7-merged dev/3.0.2-merged p0_c reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@kaka11chen @doris-robot @suxiaogang223 @morningman @zy-kkk @yiguolei @wm1581066 @dataroaring