Skip to content

[enhancement](memory) Add ThreadMemTrackerMgr BE UT#35518

Merged
xinyiZzz merged 1 commit intoapache:masterfrom
xinyiZzz:20240527_add_reserve_ut
May 29, 2024
Merged

[enhancement](memory) Add ThreadMemTrackerMgr BE UT#35518
xinyiZzz merged 1 commit intoapache:masterfrom
xinyiZzz:20240527_add_reserve_ut

Conversation

@xinyiZzz
Copy link
Copy Markdown
Contributor

@xinyiZzz xinyiZzz commented May 28, 2024

Proposed changes

For thread context tracking memory

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Copy Markdown
Contributor Author

xinyiZzz commented May 28, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 28, 2024

TPC-H: Total hot run time: 40216 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e090cf3a41e159f65c1668eecf07caa71fd78daf, data reload: false

------ Round 1 ----------------------------------
q1	18268	4437	4417	4417
q2	2485	242	199	199
q3	11972	1227	1288	1227
q4	10513	853	805	805
q5	7578	2742	2805	2742
q6	219	129	130	129
q7	962	602	599	599
q8	9612	2083	2101	2083
q9	9054	6530	6498	6498
q10	8993	3682	3686	3682
q11	450	248	235	235
q12	430	219	217	217
q13	17763	2989	2956	2956
q14	251	214	218	214
q15	501	474	467	467
q16	513	398	378	378
q17	967	578	634	578
q18	8171	7469	7614	7469
q19	5004	1582	1477	1477
q20	652	309	309	309
q21	5070	3268	3257	3257
q22	340	283	278	278
Total cold run time: 119768 ms
Total hot run time: 40216 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4323	4200	4199	4199
q2	374	253	273	253
q3	2976	2751	2698	2698
q4	1887	1631	1587	1587
q5	5280	5298	5279	5279
q6	209	123	125	123
q7	2069	1693	1743	1693
q8	3198	3350	3318	3318
q9	8374	8323	8360	8323
q10	3935	3728	3708	3708
q11	572	492	499	492
q12	781	597	610	597
q13	17573	2961	2967	2961
q14	285	264	259	259
q15	512	491	479	479
q16	473	425	404	404
q17	1780	1486	1485	1485
q18	7747	7409	7464	7409
q19	1694	1547	1595	1547
q20	2019	1785	1792	1785
q21	4854	4676	4763	4676
q22	599	508	494	494
Total cold run time: 71514 ms
Total hot run time: 53769 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 28, 2024

TPC-DS: Total hot run time: 170283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e090cf3a41e159f65c1668eecf07caa71fd78daf, data reload: false

query1	933	383	383	383
query2	6454	2457	2316	2316
query3	6650	209	202	202
query4	19308	17530	17373	17373
query5	4162	425	412	412
query6	243	153	153	153
query7	4588	297	294	294
query8	242	187	188	187
query9	8854	2449	2478	2449
query10	461	295	266	266
query11	10741	10294	10268	10268
query12	139	88	88	88
query13	1644	379	372	372
query14	9351	7621	7564	7564
query15	247	159	166	159
query16	8091	266	256	256
query17	1849	551	511	511
query18	2082	272	277	272
query19	202	155	151	151
query20	93	84	85	84
query21	198	132	130	130
query22	4558	4176	4164	4164
query23	33510	33059	32546	32546
query24	12118	2901	2834	2834
query25	639	350	352	350
query26	1752	154	155	154
query27	3029	323	322	322
query28	7486	2087	2089	2087
query29	1028	609	590	590
query30	280	148	156	148
query31	998	742	747	742
query32	93	51	55	51
query33	760	275	262	262
query34	1057	471	479	471
query35	736	618	605	605
query36	1075	938	904	904
query37	271	68	66	66
query38	2885	2800	2751	2751
query39	850	782	794	782
query40	271	122	122	122
query41	49	44	45	44
query42	104	94	94	94
query43	608	554	563	554
query44	1272	729	756	729
query45	184	160	163	160
query46	1091	700	712	700
query47	1830	1754	1771	1754
query48	376	304	298	298
query49	1197	373	377	373
query50	776	380	384	380
query51	6968	6913	6775	6775
query52	104	90	90	90
query53	346	286	285	285
query54	942	438	430	430
query55	76	73	70	70
query56	287	232	241	232
query57	1103	1066	1027	1027
query58	247	211	229	211
query59	3263	3126	3148	3126
query60	292	254	255	254
query61	90	88	89	88
query62	663	451	461	451
query63	303	288	280	280
query64	9841	2269	1716	1716
query65	3214	3086	3150	3086
query66	1385	323	322	322
query67	15248	14979	14926	14926
query68	4512	554	549	549
query69	430	272	269	269
query70	1187	1092	1159	1092
query71	420	257	274	257
query72	7874	2684	2542	2542
query73	727	325	328	325
query74	6062	5692	5631	5631
query75	3475	2608	2644	2608
query76	2940	1063	954	954
query77	439	267	262	262
query78	10340	9838	9901	9838
query79	2400	498	513	498
query80	1061	433	432	432
query81	504	231	219	219
query82	1155	93	93	93
query83	231	174	170	170
query84	238	88	85	85
query85	1299	270	259	259
query86	454	323	327	323
query87	3318	3156	3124	3124
query88	4159	2440	2440	2440
query89	456	384	366	366
query90	1965	192	193	192
query91	132	108	104	104
query92	58	120	51	51
query93	1757	526	508	508
query94	1239	190	183	183
query95	401	307	317	307
query96	599	269	273	269
query97	3193	2991	3071	2991
query98	236	216	215	215
query99	1069	862	831	831
Total cold run time: 276117 ms
Total hot run time: 170283 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 28, 2024

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e090cf3a41e159f65c1668eecf07caa71fd78daf, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.04
query4	1.68	0.07	0.06
query5	0.53	0.49	0.50
query6	1.14	0.73	0.71
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.54	0.48	0.50
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.75	0.76	0.80
query15	0.81	0.81	0.80
query16	0.35	0.36	0.37
query17	0.99	1.00	0.92
query18	0.22	0.24	0.26
query19	1.89	1.66	1.69
query20	0.02	0.01	0.01
query21	15.70	0.66	0.66
query22	4.76	6.91	1.79
query23	18.26	1.28	1.30
query24	1.44	0.33	0.26
query25	0.14	0.08	0.08
query26	0.25	0.15	0.16
query27	0.07	0.08	0.08
query28	13.32	1.01	1.00
query29	12.76	3.29	3.27
query30	0.24	0.06	0.06
query31	2.86	0.38	0.36
query32	3.30	0.47	0.46
query33	2.88	2.90	2.92
query34	17.21	4.47	4.40
query35	4.50	4.47	4.53
query36	0.66	0.47	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.85 s
Total hot run time: 30.26 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented May 28, 2024

TeamCity be ut coverage result:
Function Coverage: 35.87% (9031/25176)
Line Coverage: 27.47% (74766/272147)
Region Coverage: 26.66% (38662/145001)
Branch Coverage: 23.52% (19722/83838)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e090cf3a41e159f65c1668eecf07caa71fd78daf_e090cf3a41e159f65c1668eecf07caa71fd78daf/report/index.html

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 28, 2024

PR approved by anyone and no changes requested.

mrhhsg
mrhhsg approved these changes May 29, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@xinyiZzz xinyiZzz merged commit adacbc3 into apache:master May 29, 2024
dataroaring pushed a commit that referenced this pull request May 31, 2024
@xinyiZzz @dataroaring
[enhancement](memory) Add ThreadMemTrackerMgr BE UT (#35518)
7be7b37 
For thread context tracking memory
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 11, 2024
@xinyiZzz
[enhancement](memory) Add ThreadMemTrackerMgr BE UT (apache#35518)
d46998a 
For thread context tracking memory
@xinyiZzz xinyiZzz mentioned this pull request Jul 11, 2024
Merged
HappenLee pushed a commit that referenced this pull request Jul 11, 2024
@xinyiZzz
[branch-2.1](memory) Add ThreadMemTrackerMgr BE UT (#37654)
62e0230 
## Proposed changes

pick #35518
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrhhsg mrhhsg mrhhsg approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xinyiZzz @doris-robot @mrhhsg @yiguolei @wm1581066 @dataroaring