[feature](Recycler) Add rate limit for all obj storage client implementation by ByteYue · Pull Request #37663 · apache/doris

ByteYue · 2024-07-11T08:35:06Z

We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client.

doris-robot · 2024-07-11T08:35:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

ByteYue · 2024-07-11T08:36:54Z

run buildall

github-actions · 2024-07-11T08:40:40Z

clang-tidy review says "All clean, LGTM! 👍"

gavinchou

BE may use the same rate limiter?

ByteYue · 2024-07-13T15:30:02Z

BE may use the same rate limiter?

Yes. I'll add it into BE later.

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

ByteYue · 2024-07-14T06:05:04Z

run buildall

ByteYue · 2024-07-14T16:06:52Z

run buildall

ByteYue · 2024-07-15T03:37:39Z

run buildall

doris-robot · 2024-07-15T04:51:41Z

TPC-H: Total hot run time: 40327 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

------ Round 1 ----------------------------------
q1	17823	4528	4384	4384
q2	2545	201	195	195
q3	11792	1195	1190	1190
q4	10306	800	738	738
q5	7598	2750	2778	2750
q6	226	147	150	147
q7	971	621	612	612
q8	9480	2110	2110	2110
q9	8789	6606	6599	6599
q10	8835	3830	3756	3756
q11	448	238	239	238
q12	395	231	232	231
q13	17868	2993	2997	2993
q14	289	238	239	238
q15	530	495	488	488
q16	508	384	380	380
q17	995	646	663	646
q18	8247	7503	7403	7403
q19	6791	1422	1463	1422
q20	704	334	341	334
q21	4930	3181	3317	3181
q22	371	292	293	292
Total cold run time: 120441 ms
Total hot run time: 40327 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4346	4261	4285	4261
q2	375	276	266	266
q3	3092	2760	2769	2760
q4	1905	1567	1634	1567
q5	5304	5322	5349	5322
q6	220	134	134	134
q7	2169	1707	1738	1707
q8	3209	3395	3346	3346
q9	8503	8413	8435	8413
q10	3909	3727	3743	3727
q11	607	487	488	487
q12	775	624	606	606
q13	16903	3009	3006	3006
q14	299	271	275	271
q15	523	487	494	487
q16	491	422	435	422
q17	1773	1502	1456	1456
q18	7669	7512	7396	7396
q19	1712	1611	1449	1449
q20	2015	1828	1801	1801
q21	4846	4822	4647	4647
q22	575	495	494	494
Total cold run time: 71220 ms
Total hot run time: 54025 ms

doris-robot · 2024-07-15T05:03:12Z

TPC-DS: Total hot run time: 173782 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

query1	910	365	367	365
query2	6452	1885	1864	1864
query3	6672	210	211	210
query4	28266	17649	17403	17403
query5	4180	521	490	490
query6	255	181	165	165
query7	4594	299	288	288
query8	242	197	183	183
query9	8590	2466	2427	2427
query10	446	309	274	274
query11	11715	10114	10041	10041
query12	130	85	80	80
query13	1626	362	375	362
query14	10388	7809	7819	7809
query15	220	171	167	167
query16	7605	314	315	314
query17	1742	536	533	533
query18	1255	274	275	274
query19	192	150	150	150
query20	91	84	83	83
query21	210	130	120	120
query22	4364	4102	3946	3946
query23	33776	33206	33176	33176
query24	11991	2841	2848	2841
query25	633	365	363	363
query26	1637	150	150	150
query27	2918	277	274	274
query28	7703	2044	2062	2044
query29	1026	627	607	607
query30	283	149	148	148
query31	951	743	760	743
query32	96	53	55	53
query33	780	296	290	290
query34	967	492	492	492
query35	695	609	568	568
query36	1130	944	944	944
query37	153	81	83	81
query38	2886	2720	2730	2720
query39	884	814	808	808
query40	279	132	127	127
query41	54	50	47	47
query42	122	96	107	96
query43	505	470	479	470
query44	1199	755	737	737
query45	194	167	163	163
query46	1092	763	736	736
query47	1857	1772	1783	1772
query48	375	293	298	293
query49	1211	445	431	431
query50	803	402	402	402
query51	6865	6860	6795	6795
query52	104	96	101	96
query53	357	297	311	297
query54	1050	462	474	462
query55	80	75	78	75
query56	305	296	288	288
query57	1161	1059	1078	1059
query58	274	249	267	249
query59	3037	2600	2564	2564
query60	333	299	304	299
query61	116	114	113	113
query62	825	664	659	659
query63	334	301	293	293
query64	10665	2308	1807	1807
query65	3318	3120	3113	3113
query66	1286	346	368	346
query67	15757	15128	15020	15020
query68	4647	566	562	562
query69	624	466	358	358
query70	1232	1156	1052	1052
query71	436	296	289	289
query72	8578	6208	5647	5647
query73	758	342	327	327
query74	6156	5640	5727	5640
query75	3719	2741	2674	2674
query76	3036	921	960	921
query77	674	328	321	321
query78	9623	9113	9053	9053
query79	9109	537	536	536
query80	1858	520	495	495
query81	583	225	222	222
query82	1631	139	134	134
query83	358	169	174	169
query84	275	83	89	83
query85	2049	308	298	298
query86	446	334	320	320
query87	3332	3078	3114	3078
query88	5568	2393	2393	2393
query89	527	405	386	386
query90	1813	198	197	197
query91	133	99	104	99
query92	63	51	52	51
query93	6715	527	519	519
query94	1097	219	225	219
query95	414	322	333	322
query96	620	273	279	273
query97	3194	3086	3041	3041
query98	232	204	196	196
query99	1562	1248	1267	1248
Total cold run time: 303274 ms
Total hot run time: 173782 ms

doris-robot · 2024-07-15T05:08:43Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.49	0.50
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.52	0.55
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.60	0.59	0.59
query14	0.79	0.77	0.79
query15	0.86	0.82	0.81
query16	0.34	0.35	0.37
query17	0.98	0.98	1.01
query18	0.24	0.23	0.23
query19	1.86	1.74	1.72
query20	0.02	0.01	0.01
query21	15.41	0.73	0.65
query22	4.00	7.53	1.88
query23	18.25	1.26	1.21
query24	2.10	0.23	0.23
query25	0.15	0.09	0.09
query26	0.28	0.20	0.20
query27	0.46	0.23	0.23
query28	13.20	1.02	1.01
query29	12.58	3.36	3.35
query30	0.25	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.49	0.47
query33	2.90	2.85	2.94
query34	16.88	4.35	4.37
query35	4.43	4.44	4.40
query36	0.66	0.46	0.50
query37	0.20	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.15	0.12	0.11
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.25 s
Total hot run time: 30.56 s

github-actions · 2024-07-16T08:02:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-16T08:02:39Z

PR approved by anyone and no changes requested.

…ntation (apache#37663) We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client.

…ntation (#37663) We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client.

rate limit for obj storage

b1f12a9

gavinchou reviewed Jul 13, 2024

View reviewed changes

ByteYue added 4 commits July 14, 2024 00:38

move to common

fa01c1a

extract

af98cf5

ok

318db3a

format

08088da

github-actions bot reviewed Jul 14, 2024

View reviewed changes

Comment thread cloud/test/s3_rate_limiter_test.cpp

github-actions bot reviewed Jul 14, 2024

View reviewed changes

Comment thread cloud/test/s3_rate_limiter_test.cpp

ByteYue added 2 commits July 14, 2024 13:28

comment

a4dc51e

ok

5107cdf

use release

4185e80

ByteYue requested a review from gavinchou July 14, 2024 16:07

gavinchou reviewed Jul 14, 2024

View reviewed changes

Comment thread common/cpp/s3_rate_limiter.cpp

Comment thread common/cpp/s3_rate_limiter.cpp

log if too many times

ba800d8

ByteYue requested a review from gavinchou July 15, 2024 09:12

platoneko approved these changes Jul 16, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024

github-actions bot added the reviewed label Jul 16, 2024

gavinchou approved these changes Jul 16, 2024

View reviewed changes

gavinchou merged commit 0ad6289 into apache:master Jul 16, 2024

gavinchou added the dev/3.0.x label Jul 16, 2024

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Jul 25, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Conversation

ByteYue commented Jul 11, 2024 • edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 11, 2024

Uh oh!

ByteYue commented Jul 11, 2024

Uh oh!

github-actions bot commented Jul 11, 2024

Uh oh!

gavinchou left a comment

Choose a reason for hiding this comment

Uh oh!

ByteYue commented Jul 13, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ByteYue commented Jul 14, 2024

Uh oh!

ByteYue commented Jul 14, 2024

Uh oh!

Uh oh!

Uh oh!

ByteYue commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

github-actions bot commented Jul 16, 2024

Uh oh!

github-actions bot commented Jul 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ByteYue commented Jul 11, 2024 •

edited by gavinchou

Loading