Skip to content

[fix](cloud) MS limit max aborted txn num for the same txn label#40414

Merged
gavinchou merged 1 commit intoapache:masterfrom
swjtu-zhanglei:master-fix1
Sep 7, 2024
Merged

[fix](cloud) MS limit max aborted txn num for the same txn label#40414
gavinchou merged 1 commit intoapache:masterfrom
swjtu-zhanglei:master-fix1

Conversation

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Sep 5, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

swjtu-zhanglei commented Sep 5, 2024

run buildall

2 similar comments
@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

swjtu-zhanglei commented Sep 5, 2024

run buildall

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

swjtu-zhanglei commented Sep 5, 2024

run buildall

dataroaring
dataroaring reviewed Sep 5, 2024
View reviewed changes
Comment thread regression-test/suites/load_p0/stream_load/test_steam_load_with_data_quality.groovy

// meta-service max_num_aborted_txn is 100
for (int i = 0; i < 100; i++) {
streamLoad {
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring Sep 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100 should be got from configs of be via http. otherwise case is not stable.

Copy link
Copy Markdown
Contributor Author

@swjtu-zhanglei swjtu-zhanglei Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100 should be got from configs of be via http. otherwise case is not stable.

ms not provider http api for querying. ms's config

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

swjtu-zhanglei commented Sep 6, 2024

run buildall

swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 6, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
aabb672 
* refer to apache#40414
@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

swjtu-zhanglei commented Sep 6, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

TPC-H: Total hot run time: 38534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bbffffc1d5dfa79150034aedc0a8fb8d84550660, data reload: false

------ Round 1 ----------------------------------
q1	17758	4451	4276	4276
q2	2022	197	185	185
q3	10722	1147	1074	1074
q4	10651	813	745	745
q5	8637	2852	2868	2852
q6	236	141	142	141
q7	994	657	607	607
q8	9749	2090	2076	2076
q9	7417	6636	6640	6636
q10	7398	2231	2235	2231
q11	493	246	237	237
q12	439	231	228	228
q13	17783	3125	3068	3068
q14	281	239	243	239
q15	541	503	484	484
q16	540	433	432	432
q17	986	716	739	716
q18	7299	6902	6897	6897
q19	1449	1074	1089	1074
q20	661	350	352	350
q21	3935	2972	3001	2972
q22	1112	1053	1014	1014
Total cold run time: 111103 ms
Total hot run time: 38534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4426	4288	4282	4282
q2	372	269	264	264
q3	2877	2666	2667	2666
q4	1910	1634	1648	1634
q5	5646	5692	5714	5692
q6	239	136	132	132
q7	2177	1894	1838	1838
q8	3263	3406	3489	3406
q9	8894	8840	8664	8664
q10	3574	3324	3289	3289
q11	615	515	519	515
q12	824	644	653	644
q13	11795	3276	3284	3276
q14	319	285	286	285
q15	527	492	482	482
q16	554	506	512	506
q17	1888	1554	1505	1505
q18	8245	7858	7901	7858
q19	1743	1558	1415	1415
q20	2156	1878	1940	1878
q21	5544	5304	5440	5304
q22	1179	1050	1092	1050
Total cold run time: 68767 ms
Total hot run time: 56585 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

TPC-DS: Total hot run time: 191372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bbffffc1d5dfa79150034aedc0a8fb8d84550660, data reload: false

query1	1254	882	861	861
query2	6351	1850	1835	1835
query3	10609	4115	4010	4010
query4	58860	24450	23392	23392
query5	5298	499	514	499
query6	401	175	156	156
query7	5751	307	283	283
query8	303	221	208	208
query9	8974	2494	2461	2461
query10	511	297	260	260
query11	18196	14990	15251	14990
query12	167	109	107	107
query13	1597	392	381	381
query14	10542	6950	7097	6950
query15	247	165	174	165
query16	7589	438	470	438
query17	1128	588	565	565
query18	2034	295	285	285
query19	303	152	150	150
query20	123	107	117	107
query21	215	107	102	102
query22	4636	4490	4473	4473
query23	34482	33430	33174	33174
query24	6010	2863	2803	2803
query25	528	374	376	374
query26	690	152	153	152
query27	1808	270	268	268
query28	3656	2041	2029	2029
query29	712	398	401	398
query30	235	151	155	151
query31	908	746	764	746
query32	83	54	53	53
query33	453	281	277	277
query34	863	468	455	455
query35	833	708	721	708
query36	1078	884	936	884
query37	140	90	83	83
query38	3913	3922	3813	3813
query39	1448	1395	1421	1395
query40	200	113	115	113
query41	48	48	49	48
query42	112	95	96	95
query43	497	470	474	470
query44	1072	746	739	739
query45	196	168	173	168
query46	1096	753	785	753
query47	1910	1765	1815	1765
query48	375	293	295	293
query49	780	441	450	441
query50	823	410	462	410
query51	7039	6896	6796	6796
query52	98	86	86	86
query53	246	176	174	174
query54	560	448	445	445
query55	76	75	73	73
query56	269	250	249	249
query57	1207	1107	1078	1078
query58	215	227	240	227
query59	2872	2942	2665	2665
query60	289	262	266	262
query61	100	97	100	97
query62	818	646	646	646
query63	220	181	182	181
query64	2840	679	669	669
query65	3228	3124	3162	3124
query66	676	330	342	330
query67	15410	15395	15181	15181
query68	2982	584	585	584
query69	401	283	271	271
query70	1192	1008	1051	1008
query71	355	281	278	278
query72	5361	4051	3988	3988
query73	742	323	320	320
query74	9042	8877	8913	8877
query75	3354	2689	2720	2689
query76	1393	1033	949	949
query77	526	315	314	314
query78	9603	9020	9000	9000
query79	1028	541	522	522
query80	703	502	502	502
query81	461	229	229	229
query82	237	132	139	132
query83	171	145	149	145
query84	250	79	72	72
query85	664	295	276	276
query86	315	292	290	290
query87	4472	4127	4285	4127
query88	2885	2326	2357	2326
query89	373	279	287	279
query90	1924	188	185	185
query91	122	99	99	99
query92	59	50	50	50
query93	1054	532	530	530
query94	702	280	293	280
query95	310	246	243	243
query96	581	261	263	261
query97	3178	3084	3064	3064
query98	214	209	202	202
query99	1521	1261	1263	1261
Total cold run time: 302376 ms
Total hot run time: 191372 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

ClickBench: Total hot run time: 30.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bbffffc1d5dfa79150034aedc0a8fb8d84550660, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.09
query5	0.50	0.49	0.51
query6	1.12	0.74	0.72
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.54	0.50	0.49
query10	0.54	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	1.40	1.43	1.42
query15	0.84	0.83	0.84
query16	0.37	0.38	0.38
query17	1.01	1.08	1.04
query18	0.21	0.21	0.21
query19	1.87	1.78	1.82
query20	0.01	0.02	0.01
query21	15.39	0.66	0.66
query22	4.85	7.87	1.27
query23	18.27	1.46	1.33
query24	2.16	0.23	0.22
query25	0.15	0.08	0.07
query26	0.27	0.18	0.18
query27	0.07	0.08	0.09
query28	13.18	1.02	1.00
query29	12.65	3.32	3.35
query30	0.25	0.06	0.06
query31	2.89	0.40	0.38
query32	3.27	0.47	0.47
query33	2.99	2.98	3.03
query34	17.15	4.35	4.40
query35	4.41	4.49	4.42
query36	0.66	0.49	0.50
query37	0.19	0.17	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.82 s
Total hot run time: 30.89 s

dataroaring
dataroaring approved these changes Sep 7, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 7, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 7, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 390ef1d into apache:master Sep 7, 2024
@dataroaring dataroaring added the usercase Important user case type label label Sep 8, 2024
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 10, 2024
@swjtu-zhanglei
[fix](cloud) MS limit max aborted txn num for the same txn label (apa…
70f960e 
…che#40414)
@swjtu-zhanglei swjtu-zhanglei mentioned this pull request Sep 10, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 10, 2024
@swjtu-zhanglei
[fix](cloud) MS limit max aborted txn num for the same txn label (#40414
6d38138 
) (#40606)
gavinchou pushed a commit that referenced this pull request Sep 10, 2024
@swjtu-zhanglei @gavinchou
[fix](cloud) MS limit max aborted txn num for the same txn label (#40414
8e332ba 
)
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 12, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
0e6f00f 
* refer to apache#40414
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 13, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
dde5493 
* refer to apache#40414
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 14, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
8c715b3 
* refer to apache#40414
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 14, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
1c0d218 
* refer to apache#40414
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 18, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
7c51a89 
* refer to apache#40414

refact
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 24, 2024
@swjtu-zhanglei
[feature](cloud) Add more ut and bvar for txn lazy commit
3ba4d59 
* refer to apache#40414

refact
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 12, 2024
@gavinchou
(selectdb-cloud) Fix clean label for stream load due to null jobs
d52ccb0 
This is a part of apache#40414
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@swjtu-zhanglei @doris-robot @gavinchou @dataroaring