Skip to content

[Opt](cloud) Using file cache disks number to determine the size of the flush memtable thread pool#38789

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
liaoxin01:opt_cloud_flush
Aug 5, 2024
Merged

[Opt](cloud) Using file cache disks number to determine the size of the flush memtable thread pool#38789
liaoxin01 merged 1 commit intoapache:masterfrom
liaoxin01:opt_cloud_flush

Conversation

@liaoxin01
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 commented Aug 2, 2024 •
edited
Loading

No description provided.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 2, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Copy Markdown
Contributor Author

liaoxin01 commented Aug 2, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 2, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 2, 2024

TPC-H: Total hot run time: 41363 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7559736fd98cc45bfcbc34f516bf5775461603d7, data reload: false

------ Round 1 ----------------------------------
q1	17605	4146	4059	4059
q2	2019	199	198	198
q3	10563	1322	1361	1322
q4	10175	807	970	807
q5	7614	2930	2975	2930
q6	218	135	136	135
q7	1065	614	612	612
q8	9438	1841	1928	1841
q9	8547	6608	6588	6588
q10	8737	3827	3814	3814
q11	434	247	241	241
q12	438	229	224	224
q13	17771	2947	2961	2947
q14	268	242	247	242
q15	519	477	502	477
q16	526	396	394	394
q17	972	949	905	905
q18	7945	7325	7193	7193
q19	1855	1215	1204	1204
q20	558	316	343	316
q21	5287	4637	4772	4637
q22	346	277	281	277
Total cold run time: 112900 ms
Total hot run time: 41363 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4123	4022	4012	4012
q2	328	220	219	219
q3	2986	3006	3135	3006
q4	1992	1997	1976	1976
q5	5593	5462	5469	5462
q6	228	130	131	130
q7	2102	1758	1814	1758
q8	3291	3370	3332	3332
q9	8650	8606	8704	8606
q10	3978	4103	3967	3967
q11	548	448	442	442
q12	747	597	636	597
q13	16560	3114	3097	3097
q14	313	250	275	250
q15	543	489	487	487
q16	461	394	405	394
q17	1755	1753	1733	1733
q18	8218	7647	7709	7647
q19	1725	1713	1700	1700
q20	2109	1869	1834	1834
q21	5639	5485	5345	5345
q22	520	449	465	449
Total cold run time: 72409 ms
Total hot run time: 56443 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 2, 2024

TPC-DS: Total hot run time: 169567 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7559736fd98cc45bfcbc34f516bf5775461603d7, data reload: false

query1	915	371	373	371
query2	6467	1649	1689	1649
query3	6666	216	225	216
query4	19569	17370	17104	17104
query5	3658	524	530	524
query6	288	170	168	168
query7	4602	306	298	298
query8	255	182	191	182
query9	8536	2360	2340	2340
query10	426	276	271	271
query11	10293	10012	9901	9901
query12	120	95	85	85
query13	1628	387	382	382
query14	8381	7618	7278	7278
query15	201	163	178	163
query16	6797	452	458	452
query17	957	566	547	547
query18	1852	279	276	276
query19	188	142	143	142
query20	89	82	87	82
query21	205	102	100	100
query22	4165	3927	3953	3927
query23	33546	33595	33573	33573
query24	10234	3076	3109	3076
query25	700	413	421	413
query26	1752	153	152	152
query27	2959	280	289	280
query28	7654	2011	1990	1990
query29	1295	433	454	433
query30	238	157	150	150
query31	934	794	756	756
query32	100	57	56	56
query33	697	319	331	319
query34	913	504	511	504
query35	848	798	770	770
query36	1047	900	880	880
query37	290	88	89	88
query38	2981	2843	2832	2832
query39	910	856	828	828
query40	290	115	113	113
query41	47	46	43	43
query42	118	96	100	96
query43	457	428	420	420
query44	1206	749	738	738
query45	209	175	180	175
query46	1087	846	784	784
query47	1794	1699	1688	1688
query48	367	300	297	297
query49	1009	426	433	426
query50	903	439	456	439
query51	6810	6722	6742	6722
query52	100	94	88	88
query53	270	187	181	181
query54	640	481	458	458
query55	75	78	74	74
query56	278	254	274	254
query57	1102	1020	1025	1020
query58	260	256	262	256
query59	2566	2309	2288	2288
query60	304	305	268	268
query61	93	95	96	95
query62	881	669	662	662
query63	231	199	183	183
query64	5689	1906	1899	1899
query65	3183	3076	3111	3076
query66	1312	339	328	328
query67	15282	14762	14787	14762
query68	4255	588	637	588
query69	440	311	295	295
query70	1125	1040	1027	1027
query71	431	289	283	283
query72	7055	2651	2555	2555
query73	787	333	332	332
query74	6017	5618	5678	5618
query75	3375	2710	2684	2684
query76	2388	1213	1301	1213
query77	449	316	311	311
query78	9262	8998	8817	8817
query79	1890	539	555	539
query80	1096	523	521	521
query81	539	228	228	228
query82	1039	133	131	131
query83	240	172	177	172
query84	272	80	81	80
query85	1222	343	302	302
query86	420	310	278	278
query87	3241	3076	3087	3076
query88	3067	2502	2546	2502
query89	390	293	294	293
query90	1728	203	201	201
query91	137	177	100	100
query92	58	52	50	50
query93	1513	613	618	613
query94	783	298	294	294
query95	389	274	287	274
query96	605	294	284	284
query97	3181	3108	3056	3056
query98	217	211	195	195
query99	1637	1294	1264	1264
Total cold run time: 259167 ms
Total hot run time: 169567 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 2, 2024

ClickBench: Total hot run time: 30.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7559736fd98cc45bfcbc34f516bf5775461603d7, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.08	0.07
query5	0.50	0.48	0.48
query6	1.15	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.52
query10	0.57	0.57	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.62	0.61	0.60
query14	0.79	0.78	0.80
query15	0.90	0.87	0.87
query16	0.35	0.36	0.34
query17	0.99	1.03	1.00
query18	0.23	0.21	0.20
query19	1.83	1.73	1.80
query20	0.02	0.01	0.01
query21	15.40	0.76	0.65
query22	4.17	7.68	1.28
query23	17.83	1.40	1.38
query24	2.31	0.22	0.22
query25	0.19	0.08	0.08
query26	0.32	0.21	0.21
query27	0.47	0.22	0.23
query28	13.16	0.98	0.98
query29	12.52	3.28	3.26
query30	0.26	0.06	0.06
query31	2.84	0.41	0.41
query32	3.23	0.48	0.50
query33	2.88	2.99	2.95
query34	15.41	4.27	4.23
query35	4.29	4.28	4.29
query36	0.68	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.03
query40	0.16	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.61 s
Total hot run time: 30.03 s

dataroaring
dataroaring approved these changes Aug 2, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring added the usercase Important user case type label label Aug 2, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 2, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 2, 2024

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Aug 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 455018b into apache:master Aug 5, 2024
@liaoxin01 liaoxin01 deleted the opt_cloud_flush branch August 5, 2024 02:25
dataroaring pushed a commit that referenced this pull request Aug 5, 2024
@liaoxin01 @dataroaring
[Opt](cloud) Using file cache disks number to determine the size of t…
ebd3525 
…he flush memtable thread pool (#38789)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@0AyanamiRei 0AyanamiRei mentioned this pull request Jan 8, 2026
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Jan 20, 2026
@0AyanamiRei
[fix](flush) Fix the issue of incorrect calculation of the number of …
3a86ab6 
…flush threads in cloud mode (#59524)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx #38789
github-actions bot pushed a commit that referenced this pull request Jan 20, 2026
@0AyanamiRei
[fix](flush) Fix the issue of incorrect calculation of the number of …
1d7e99f 
…flush threads in cloud mode (#59524)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx #38789
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.1-merged doing reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01 @doris-robot @sollhui @dataroaring