Skip to content

[fix](memory) Fix query mem tracker log when destruction not equal to 0#34901

Merged
yiguolei merged 1 commit intoapache:masterfrom
xinyiZzz:20240515_fix_memtracker
May 16, 2024
Merged

[fix](memory) Fix query mem tracker log when destruction not equal to 0#34901
yiguolei merged 1 commit intoapache:masterfrom
xinyiZzz:20240515_fix_memtracker

Conversation

@xinyiZzz
Copy link
Copy Markdown
Contributor

@xinyiZzz xinyiZzz commented May 15, 2024

Proposed changes

Fix query mem tracker log when destruction not equal to 0

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Copy Markdown
Contributor Author

xinyiZzz commented May 15, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 41841 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

------ Round 1 ----------------------------------
q1	17588	4294	4235	4235
q2	2006	183	186	183
q3	10489	1200	1185	1185
q4	10199	863	862	862
q5	7475	2720	2714	2714
q6	230	130	132	130
q7	1024	603	593	593
q8	9238	2131	2077	2077
q9	9708	6713	6619	6619
q10	9069	3877	3864	3864
q11	452	235	251	235
q12	417	234	233	233
q13	17201	3283	3172	3172
q14	279	220	230	220
q15	514	481	475	475
q16	498	396	400	396
q17	985	743	682	682
q18	8190	7936	7831	7831
q19	4730	1541	1492	1492
q20	643	331	308	308
q21	5138	4143	4060	4060
q22	350	275	296	275
Total cold run time: 116423 ms
Total hot run time: 41841 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4508	4412	4361	4361
q2	372	252	272	252
q3	3202	2937	2952	2937
q4	1892	1596	1582	1582
q5	5461	5499	5491	5491
q6	218	126	129	126
q7	2291	1958	2003	1958
q8	3253	3360	3378	3360
q9	8641	8692	8684	8684
q10	3880	3760	3860	3760
q11	613	498	508	498
q12	792	634	638	634
q13	16255	3116	3179	3116
q14	309	259	267	259
q15	527	496	496	496
q16	460	410	415	410
q17	1765	1474	1459	1459
q18	7570	7624	7401	7401
q19	1632	1502	1556	1502
q20	1939	1754	1789	1754
q21	6127	4880	4835	4835
q22	570	499	497	497
Total cold run time: 72277 ms
Total hot run time: 55372 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

TeamCity be ut coverage result:
Function Coverage: 35.65% (8981/25195)
Line Coverage: 27.31% (74245/271861)
Region Coverage: 26.54% (38362/144541)
Branch Coverage: 23.36% (19566/83752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f45e796c3a8308f6abf45e96f62bceb1ff26b10_3f45e796c3a8308f6abf45e96f62bceb1ff26b10/report/index.html

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 186706 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

query1	907	464	369	369
query2	6942	2381	2479	2381
query3	6647	206	215	206
query4	22816	21235	21135	21135
query5	4124	411	412	411
query6	263	172	175	172
query7	4586	293	286	286
query8	235	193	191	191
query9	8454	2403	2391	2391
query10	436	254	258	254
query11	14603	14174	14168	14168
query12	136	89	88	88
query13	1656	377	377	377
query14	8716	7716	7585	7585
query15	214	169	175	169
query16	7927	261	257	257
query17	1737	549	551	549
query18	2038	283	269	269
query19	200	149	147	147
query20	91	85	84	84
query21	196	130	128	128
query22	5097	4869	4858	4858
query23	34106	33301	33632	33301
query24	7275	2859	2871	2859
query25	574	365	367	365
query26	702	157	152	152
query27	2190	312	329	312
query28	5319	2054	2059	2054
query29	846	616	608	608
query30	283	174	174	174
query31	964	747	746	746
query32	91	53	53	53
query33	494	252	250	250
query34	857	473	480	473
query35	765	685	674	674
query36	1024	919	925	919
query37	108	67	71	67
query38	2883	2815	2770	2770
query39	1604	1546	1561	1546
query40	189	126	124	124
query41	44	43	43	43
query42	107	92	95	92
query43	587	560	558	558
query44	1047	715	721	715
query45	265	256	225	225
query46	1064	754	713	713
query47	1968	1888	1904	1888
query48	363	293	293	293
query49	873	387	390	387
query50	754	377	380	377
query51	6901	6824	6792	6792
query52	103	90	100	90
query53	350	287	276	276
query54	524	434	428	428
query55	73	70	71	70
query56	244	225	223	223
query57	1231	1168	1154	1154
query58	211	202	229	202
query59	3301	3105	3030	3030
query60	290	227	249	227
query61	91	90	88	88
query62	620	475	469	469
query63	307	278	280	278
query64	8438	7451	7409	7409
query65	3114	3071	3109	3071
query66	801	367	346	346
query67	15301	14847	15214	14847
query68	4501	540	530	530
query69	471	303	309	303
query70	1188	1145	1161	1145
query71	352	258	265	258
query72	7226	2603	2340	2340
query73	704	324	326	324
query74	6522	6079	6136	6079
query75	3254	2645	2603	2603
query76	2319	1049	959	959
query77	392	263	259	259
query78	10585	10133	10226	10133
query79	2347	507	518	507
query80	1075	436	423	423
query81	527	237	242	237
query82	721	95	93	93
query83	222	160	171	160
query84	242	88	88	88
query85	1080	270	262	262
query86	449	329	318	318
query87	3291	3074	3102	3074
query88	4172	2403	2389	2389
query89	470	373	378	373
query90	2044	183	183	183
query91	123	97	99	97
query92	60	56	48	48
query93	1701	524	501	501
query94	1137	178	180	178
query95	401	305	299	299
query96	587	280	266	266
query97	3147	2947	2960	2947
query98	250	228	218	218
query99	1165	915	899	899
Total cold run time: 269992 ms
Total hot run time: 186706 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.97	0.07	0.07
query5	0.51	0.48	0.52
query6	1.12	0.75	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.48
query10	0.55	0.55	0.54
query11	0.15	0.11	0.12
query12	0.13	0.11	0.12
query13	0.59	0.59	0.60
query14	0.77	0.78	0.77
query15	0.84	0.80	0.82
query16	0.36	0.37	0.37
query17	1.02	1.00	1.04
query18	0.21	0.26	0.23
query19	1.85	1.72	1.76
query20	0.01	0.00	0.00
query21	15.48	0.68	0.66
query22	4.20	7.15	2.14
query23	18.29	1.30	1.32
query24	1.58	0.32	0.21
query25	0.15	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.07	0.08
query28	13.33	1.02	0.98
query29	13.30	3.32	3.29
query30	0.24	0.06	0.05
query31	2.90	0.39	0.39
query32	3.29	0.45	0.46
query33	2.82	2.82	2.79
query34	17.14	4.39	4.40
query35	4.49	4.43	4.65
query36	0.66	0.47	0.48
query37	0.17	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.97 s
Total hot run time: 30.61 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 41013 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

------ Round 1 ----------------------------------
q1	18035	4542	4453	4453
q2	2865	185	200	185
q3	11390	1172	1224	1172
q4	10780	847	812	812
q5	7505	2688	2615	2615
q6	215	132	137	132
q7	994	582	568	568
q8	9229	2085	2086	2085
q9	8806	6580	6525	6525
q10	8977	3730	3726	3726
q11	450	242	237	237
q12	445	222	217	217
q13	17773	3013	3022	3013
q14	260	214	227	214
q15	505	487	475	475
q16	508	383	388	383
q17	964	658	764	658
q18	8004	7540	7582	7540
q19	6201	1534	1514	1514
q20	663	297	296	296
q21	4990	3912	4069	3912
q22	343	281	289	281
Total cold run time: 119902 ms
Total hot run time: 41013 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4286	4240	4240
q2	373	267	273	267
q3	2964	2812	2713	2713
q4	1936	1648	1603	1603
q5	5250	5288	5269	5269
q6	212	127	130	127
q7	2211	1854	1839	1839
q8	3239	3324	3371	3324
q9	8234	8307	8385	8307
q10	3903	3646	3677	3646
q11	569	496	484	484
q12	735	568	646	568
q13	17324	2945	2947	2945
q14	293	270	260	260
q15	527	473	474	473
q16	475	402	430	402
q17	1777	1486	1476	1476
q18	7725	7553	7448	7448
q19	1658	1564	1522	1522
q20	1994	1763	1794	1763
q21	5063	4828	4863	4828
q22	596	496	510	496
Total cold run time: 71456 ms
Total hot run time: 54000 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 186551 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

query1	905	387	374	374
query2	6545	2441	2351	2351
query3	6645	218	219	218
query4	22841	21085	21187	21085
query5	4198	452	424	424
query6	259	175	175	175
query7	4589	295	297	295
query8	243	194	183	183
query9	8731	2404	2395	2395
query10	458	255	262	255
query11	14701	14261	14157	14157
query12	141	95	90	90
query13	1655	387	379	379
query14	10031	7117	8496	7117
query15	280	169	172	169
query16	8161	269	264	264
query17	1842	559	532	532
query18	2061	278	273	273
query19	215	149	150	149
query20	93	83	87	83
query21	193	133	131	131
query22	5098	4872	4887	4872
query23	34266	33551	33612	33551
query24	12143	2854	2925	2854
query25	632	383	365	365
query26	1753	160	160	160
query27	3161	333	335	333
query28	7691	2079	2064	2064
query29	997	618	592	592
query30	302	174	181	174
query31	984	787	754	754
query32	92	55	53	53
query33	749	244	242	242
query34	1107	490	484	484
query35	800	687	676	676
query36	1076	943	935	935
query37	277	70	71	70
query38	2952	2797	2788	2788
query39	1637	1576	1526	1526
query40	274	125	125	125
query41	48	43	42	42
query42	101	95	98	95
query43	614	534	577	534
query44	1233	738	763	738
query45	270	249	256	249
query46	1076	750	714	714
query47	1963	1894	1863	1863
query48	375	303	318	303
query49	1184	394	391	391
query50	783	391	383	383
query51	6812	6794	6760	6760
query52	99	91	94	91
query53	348	287	293	287
query54	871	440	418	418
query55	74	75	75	75
query56	234	219	222	219
query57	1261	1155	1138	1138
query58	217	194	216	194
query59	3676	3220	3029	3029
query60	250	228	232	228
query61	89	85	107	85
query62	696	462	474	462
query63	310	293	283	283
query64	9750	7427	7352	7352
query65	3139	3110	3079	3079
query66	1327	345	334	334
query67	15487	14925	15191	14925
query68	4633	531	534	531
query69	513	310	308	308
query70	1177	1031	1106	1031
query71	412	259	287	259
query72	7068	2558	2324	2324
query73	720	325	320	320
query74	6488	6202	6119	6119
query75	3382	2647	2643	2643
query76	2828	1068	996	996
query77	385	269	269	269
query78	10505	10160	10063	10063
query79	2394	533	528	528
query80	1041	442	438	438
query81	521	242	236	236
query82	1135	96	95	95
query83	245	164	168	164
query84	241	88	83	83
query85	1433	366	264	264
query86	442	327	322	322
query87	3275	3146	3146	3146
query88	4403	2361	2372	2361
query89	468	379	399	379
query90	1945	194	188	188
query91	124	96	97	96
query92	57	48	48	48
query93	1733	511	495	495
query94	1259	181	189	181
query95	394	304	309	304
query96	596	264	273	264
query97	3131	2999	3012	2999
query98	237	226	211	211
query99	1121	909	898	898
Total cold run time: 287371 ms
Total hot run time: 186551 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f45e796c3a8308f6abf45e96f62bceb1ff26b10, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.08	0.07
query5	0.50	0.47	0.50
query6	1.12	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.55	0.55
query11	0.17	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.59	0.60
query14	0.79	0.78	0.77
query15	0.84	0.81	0.81
query16	0.37	0.37	0.37
query17	1.04	1.01	0.93
query18	0.22	0.23	0.24
query19	1.80	1.72	1.69
query20	0.01	0.01	0.01
query21	15.72	0.67	0.66
query22	3.94	7.18	2.44
query23	18.28	1.32	1.21
query24	1.59	0.28	0.25
query25	0.14	0.09	0.08
query26	0.27	0.16	0.16
query27	0.07	0.08	0.08
query28	13.29	1.02	1.01
query29	13.16	3.30	3.32
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.27	0.48	0.48
query33	2.86	2.83	2.84
query34	17.03	4.42	4.43
query35	4.52	4.45	4.60
query36	0.66	0.47	0.47
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.16	0.14	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.43 s
Total hot run time: 30.91 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 15, 2024

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes May 16, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz
Copy link
Copy Markdown
Contributor Author

xinyiZzz commented May 16, 2024

comment run performance

1 similar comment
@xinyiZzz
Copy link
Copy Markdown
Contributor Author

xinyiZzz commented May 16, 2024

comment run performance

@yiguolei yiguolei force-pushed the 20240515_fix_memtracker branch from 3f45e79 to f969c4b Compare May 16, 2024 13:46
@yiguolei
Copy link
Copy Markdown
Contributor

yiguolei commented May 16, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 16, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 16, 2024

TeamCity be ut coverage result:
Function Coverage: 35.70% (9003/25219)
Line Coverage: 27.37% (74494/272148)
Region Coverage: 26.61% (38513/144707)
Branch Coverage: 23.44% (19647/83836)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f969c4ba45dd182fb2b71360eb9def24082654be_f969c4ba45dd182fb2b71360eb9def24082654be/report/index.html

@yiguolei yiguolei merged commit 59b0ed3 into apache:master May 16, 2024
dataroaring pushed a commit that referenced this pull request May 26, 2024
@xinyiZzz @dataroaring
[fix](memory) Fix query mem tracker log when destruction not equal to 0
0e8e8a6 
#34901
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
@xinyiZzz
[fix](memory) Fix query mem tracker log when destruction not equal to 0
7827ffd 
apache#34901
@xinyiZzz xinyiZzz mentioned this pull request Sep 9, 2024
Closed
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
@xinyiZzz
[fix](memory) Fix query mem tracker log when destruction not equal to 0
af1fdac 
apache#34901
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
@xinyiZzz
[fix](memory) Fix query mem tracker log when destruction not equal to 0
36dbc99 
apache#34901
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
@xinyiZzz @yiguolei
[fix](memory) Fix query mem tracker log when destruction not equal to 0
2335d11 
apache#34901
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 11, 2024
@xinyiZzz @yiguolei
[fix](memory) Fix query mem tracker log when destruction not equal to 0
47fd19f 
apache#34901
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 13, 2024
@xinyiZzz
[fix](memory) Fix query mem tracker log when destruction not equal to 0
03d920e 
apache#34901
@xinyiZzz xinyiZzz mentioned this pull request Sep 13, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 14, 2024
@xinyiZzz @yiguolei @Doris-Extras
[branch-2.1](memory) Allocator support address sanitizers (#40836)
f16615a 
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@wangbo wangbo wangbo approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xinyiZzz @doris-robot @yiguolei @wangbo