[enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time by ByteYue · Pull Request #38294 · apache/doris

ByteYue · 2024-07-24T06:57:24Z

Add one bvar to monitor the failed request due to exceeds rate limit.

doris-robot · 2024-07-24T06:57:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-07-24T07:03:36Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-24T11:09:47Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-24T12:47:40Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions

clang-tidy made some suggestions

ByteYue · 2024-07-29T14:20:36Z

run buildall

github-actions · 2024-07-29T14:40:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-29T14:40:51Z

PR approved by anyone and no changes requested.

doris-robot · 2024-07-29T15:04:49Z

TPC-H: Total hot run time: 41956 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ecd971015410c28066b4d41c1d83c0033adbce1, data reload: false

------ Round 1 ----------------------------------
q1	17620	4181	4051	4051
q2	2017	204	213	204
q3	10605	1372	1365	1365
q4	10700	841	1022	841
q5	8262	3140	3054	3054
q6	225	138	137	137
q7	1061	626	631	626
q8	10438	1884	2012	1884
q9	8788	6759	6678	6678
q10	8759	3829	3858	3829
q11	438	256	251	251
q12	414	224	228	224
q13	17763	2920	2925	2920
q14	273	251	248	248
q15	526	482	484	482
q16	513	397	397	397
q17	977	929	955	929
q18	8143	7491	7297	7297
q19	1546	1221	1227	1221
q20	573	322	357	322
q21	5349	4717	4805	4717
q22	359	282	279	279
Total cold run time: 115349 ms
Total hot run time: 41956 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4095	4014	4026	4014
q2	332	218	222	218
q3	3001	3033	2974	2974
q4	1905	1892	1889	1889
q5	5277	5252	5233	5233
q6	217	128	127	127
q7	2060	1699	1672	1672
q8	3223	3324	3312	3312
q9	8344	8382	8356	8356
q10	3779	3858	3908	3858
q11	560	449	442	442
q12	713	555	546	546
q13	9777	2954	2914	2914
q14	283	255	257	255
q15	517	469	473	469
q16	442	395	397	395
q17	1732	1684	1649	1649
q18	7676	7487	7258	7258
q19	1671	1691	1676	1676
q20	1969	1763	1736	1736
q21	5485	5132	5281	5132
q22	509	430	457	430
Total cold run time: 63567 ms
Total hot run time: 54555 ms

add bvar for ms

ByteYue · 2024-07-29T15:12:31Z

run buildall

doris-robot · 2024-07-29T15:30:40Z

TPC-H: Total hot run time: 41917 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false

------ Round 1 ----------------------------------
q1	17629	4131	4120	4120
q2	2030	207	214	207
q3	10452	1251	1379	1251
q4	10157	861	961	861
q5	7659	3024	3013	3013
q6	226	141	140	140
q7	1039	619	638	619
q8	9472	1884	1943	1884
q9	9082	6634	6642	6634
q10	8805	3892	3891	3891
q11	439	256	256	256
q12	449	229	235	229
q13	17755	2923	2963	2923
q14	270	247	250	247
q15	530	488	501	488
q16	524	399	391	391
q17	986	962	906	906
q18	8032	7367	7247	7247
q19	1375	1227	1210	1210
q20	570	338	335	335
q21	5411	4782	4855	4782
q22	356	285	283	283
Total cold run time: 113248 ms
Total hot run time: 41917 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4080	4031	4023	4023
q2	327	221	220	220
q3	3014	2981	2973	2973
q4	1933	1904	1936	1904
q5	5247	5268	5261	5261
q6	223	133	130	130
q7	2105	1665	1704	1665
q8	3227	3332	3349	3332
q9	8332	8341	8327	8327
q10	3799	3871	3845	3845
q11	555	452	451	451
q12	721	550	548	548
q13	5125	2975	2987	2975
q14	299	266	259	259
q15	515	473	471	471
q16	471	406	395	395
q17	1721	1715	1701	1701
q18	7754	7416	7203	7203
q19	1688	1693	1694	1693
q20	1973	1789	1764	1764
q21	5503	5403	5216	5216
q22	505	477	463	463
Total cold run time: 59117 ms
Total hot run time: 54819 ms

doris-robot · 2024-07-29T15:41:19Z

TPC-DS: Total hot run time: 168590 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false

query1	941	373	369	369
query2	6479	1676	1665	1665
query3	6667	207	221	207
query4	20372	17445	17282	17282
query5	4325	525	512	512
query6	285	167	173	167
query7	4595	306	299	299
query8	258	193	201	193
query9	8508	2434	2391	2391
query10	439	271	255	255
query11	10486	10081	9978	9978
query12	134	89	87	87
query13	1632	373	368	368
query14	9605	7018	8478	7018
query15	200	173	160	160
query16	7086	488	450	450
query17	945	604	550	550
query18	1914	285	275	275
query19	195	144	142	142
query20	89	86	86	86
query21	206	103	105	103
query22	4333	3981	3920	3920
query23	33762	33169	32843	32843
query24	10382	3084	2998	2998
query25	685	385	391	385
query26	1802	153	153	153
query27	2996	280	279	279
query28	7074	2007	1984	1984
query29	1345	444	452	444
query30	280	151	151	151
query31	925	751	738	738
query32	103	59	57	57
query33	700	310	309	309
query34	910	480	482	480
query35	845	710	733	710
query36	1004	865	869	865
query37	312	81	80	80
query38	2859	2737	2808	2737
query39	846	812	818	812
query40	283	114	117	114
query41	48	45	48	45
query42	121	99	99	99
query43	473	397	400	397
query44	1190	717	720	717
query45	213	177	174	174
query46	1088	787	774	774
query47	1821	1725	1735	1725
query48	375	292	294	292
query49	1201	440	442	440
query50	898	434	472	434
query51	6709	6595	6664	6595
query52	103	96	90	90
query53	256	179	175	175
query54	617	452	453	452
query55	77	75	79	75
query56	282	258	254	254
query57	1136	1076	1025	1025
query58	281	261	257	257
query59	2627	2487	2280	2280
query60	301	266	283	266
query61	101	97	98	97
query62	923	658	682	658
query63	210	186	199	186
query64	5946	1914	1912	1912
query65	3146	3119	3093	3093
query66	1444	333	335	333
query67	15338	14860	14851	14851
query68	4341	562	575	562
query69	450	300	306	300
query70	1128	1025	1046	1025
query71	359	269	281	269
query72	7181	2691	2542	2542
query73	760	327	338	327
query74	6047	5631	5613	5613
query75	3343	2737	2711	2711
query76	2245	1361	1414	1361
query77	437	305	300	300
query78	9502	8979	8865	8865
query79	1649	539	524	524
query80	1201	510	496	496
query81	571	227	228	227
query82	1163	134	128	128
query83	237	169	164	164
query84	274	78	81	78
query85	1282	321	303	303
query86	394	279	310	279
query87	3259	3150	3122	3122
query88	2962	2495	2530	2495
query89	378	289	284	284
query90	1760	191	189	189
query91	133	102	104	102
query92	62	51	52	51
query93	1456	613	614	613
query94	887	305	302	302
query95	384	268	270	268
query96	601	281	280	280
query97	3161	3028	3036	3028
query98	222	204	197	197
query99	1673	1310	1273	1273
Total cold run time: 263234 ms
Total hot run time: 168590 ms

doris-robot · 2024-07-29T15:46:47Z

ClickBench: Total hot run time: 30.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.21	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.48
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.58	0.53	0.50
query10	0.57	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.77	0.78	0.82
query15	0.88	0.85	0.86
query16	0.36	0.35	0.36
query17	0.99	0.97	1.00
query18	0.22	0.21	0.21
query19	1.84	1.74	1.76
query20	0.01	0.01	0.01
query21	15.43	0.74	0.67
query22	3.78	6.70	1.51
query23	17.97	1.37	1.31
query24	2.31	0.23	0.23
query25	0.18	0.09	0.09
query26	0.31	0.21	0.22
query27	0.46	0.23	0.23
query28	13.17	0.99	0.97
query29	12.52	3.25	3.25
query30	0.25	0.05	0.05
query31	2.86	0.41	0.40
query32	3.24	0.51	0.48
query33	2.97	2.93	2.93
query34	15.44	4.26	4.26
query35	4.32	4.30	4.29
query36	0.67	0.48	0.48
query37	0.19	0.17	0.16
query38	0.17	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.44 s
Total hot run time: 30.18 s

github-actions · 2024-07-30T14:17:02Z

PR approved by at least one committer and no changes requested.

…eep time and failure time (#38294) Add one bvar to monitor the failed request due to exceeds rate limit.

…eep time and failure time (apache#38294) Add one bvar to monitor the failed request due to exceeds rate limit.

ByteYue force-pushed the rate_limit_bvar branch from c1ca982 to cee6085 Compare July 24, 2024 11:03

ByteYue force-pushed the rate_limit_bvar branch from 30dfab0 to f739e51 Compare July 25, 2024 07:28

github-actions bot reviewed Jul 25, 2024

View reviewed changes

Comment thread cloud/src/recycler/s3_accessor.h

ByteYue force-pushed the rate_limit_bvar branch from aa29ba5 to 6ecd971 Compare July 25, 2024 12:47

gavinchou changed the title ~~[enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time~~ [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time Jul 29, 2024

gavinchou reviewed Jul 29, 2024

View reviewed changes

Comment thread common/cpp/s3_rate_limiter.cpp Outdated

gavinchou previously approved these changes Jul 29, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 29, 2024

add bvar

6ac14c9

add bvar for ms

ByteYue dismissed gavinchou’s stale review via 6ac14c9 July 29, 2024 15:10

ByteYue force-pushed the rate_limit_bvar branch from 6ecd971 to 6ac14c9 Compare July 29, 2024 15:10

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 29, 2024

gavinchou added the dev/3.0.x label Jul 30, 2024

ByteYue requested a review from gavinchou July 30, 2024 05:21

gavinchou approved these changes Jul 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024

luwei16 approved these changes Jul 30, 2024

View reviewed changes

gavinchou merged commit ec2cbbc into apache:master Jul 31, 2024

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Jul 31, 2024

dataroaring pushed a commit that referenced this pull request Jul 31, 2024

[enhance](RateLimit) Add bvar to monitor object storage rate limit sl…

2aeb3b5

…eep time and failure time (#38294) Add one bvar to monitor the failed request due to exceeds rate limit.

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Conversation

ByteYue commented Jul 24, 2024 • edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 24, 2024

Uh oh!

github-actions bot commented Jul 24, 2024

Uh oh!

github-actions bot commented Jul 24, 2024

Uh oh!

github-actions bot commented Jul 24, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ByteYue commented Jul 29, 2024

Uh oh!

Uh oh!

github-actions bot commented Jul 29, 2024

Uh oh!

github-actions bot commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

ByteYue commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

github-actions bot commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ByteYue commented Jul 24, 2024 •

edited by gavinchou

Loading