Skip to content

[fix](cloud) should do check before abort transaction#40463

Merged
dataroaring merged 1 commit intoapache:masterfrom
sollhui:fix_job_schedule_error
Sep 9, 2024
Merged

[fix](cloud) should do check before abort transaction#40463
dataroaring merged 1 commit intoapache:masterfrom
sollhui:fix_job_schedule_error

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented Sep 6, 2024

When routine load task transaction is abort, it should do check before abort transaction, otherwise, it may cause concurrent modifications to the routineLoadTaskInfoList, which in the Java language may result in elements in the list being null, leading to a loop throwing NullPointerException during scheduling and making it impossible to schedule routine load task to consume Kafka stream.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Sep 6, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

TPC-H: Total hot run time: 37633 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a89ce73c3db2c5d03cb9640491dd2e90d90e4f8, data reload: false

------ Round 1 ----------------------------------
q1	17615	4285	4293	4285
q2	2022	198	192	192
q3	11677	919	1122	919
q4	10524	730	813	730
q5	7755	2820	2753	2753
q6	252	139	134	134
q7	959	614	606	606
q8	9323	2062	2082	2062
q9	6978	6526	6547	6526
q10	6990	2219	2180	2180
q11	449	241	240	240
q12	403	217	221	217
q13	17769	3114	3065	3065
q14	278	247	247	247
q15	537	507	478	478
q16	519	435	433	433
q17	981	686	803	686
q18	7338	6751	6922	6751
q19	1405	1039	972	972
q20	717	341	335	335
q21	3889	3074	2813	2813
q22	1123	1019	1009	1009
Total cold run time: 109503 ms
Total hot run time: 37633 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4317	4249	4241	4241
q2	373	278	269	269
q3	2875	2683	2668	2668
q4	1928	1652	1609	1609
q5	5514	5656	5676	5656
q6	230	135	134	134
q7	2223	1843	1833	1833
q8	3288	3434	3404	3404
q9	8806	8804	8739	8739
q10	3559	3335	3361	3335
q11	656	500	527	500
q12	819	685	663	663
q13	14225	3350	3201	3201
q14	323	291	288	288
q15	532	489	497	489
q16	559	502	491	491
q17	1822	1529	1532	1529
q18	8107	7663	7758	7663
q19	1764	1531	1688	1531
q20	2152	1902	1948	1902
q21	5732	5336	5423	5336
q22	1170	1061	1046	1046
Total cold run time: 70974 ms
Total hot run time: 56527 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

TPC-DS: Total hot run time: 192141 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a89ce73c3db2c5d03cb9640491dd2e90d90e4f8, data reload: false

query1	1292	882	909	882
query2	6309	1949	1859	1859
query3	10588	3864	4001	3864
query4	59649	26325	23221	23221
query5	5456	493	481	481
query6	400	173	162	162
query7	5768	308	307	307
query8	310	213	214	213
query9	8956	2500	2491	2491
query10	487	286	257	257
query11	17171	15111	15187	15111
query12	158	114	105	105
query13	1549	414	403	403
query14	11147	7007	7097	7007
query15	251	175	175	175
query16	7570	478	477	477
query17	1137	574	558	558
query18	2073	309	302	302
query19	301	144	139	139
query20	119	107	107	107
query21	205	109	108	108
query22	4693	4359	4642	4359
query23	34435	33274	33690	33274
query24	5932	2830	2848	2830
query25	520	396	393	393
query26	689	151	169	151
query27	1790	278	283	278
query28	3782	2053	2033	2033
query29	687	402	402	402
query30	238	154	154	154
query31	909	719	777	719
query32	76	52	53	52
query33	448	296	278	278
query34	884	461	483	461
query35	842	731	720	720
query36	1064	910	896	896
query37	140	83	83	83
query38	4008	3852	3851	3851
query39	1455	1415	1397	1397
query40	192	120	113	113
query41	48	45	45	45
query42	113	98	96	96
query43	514	486	484	484
query44	1071	742	735	735
query45	201	173	165	165
query46	1104	753	767	753
query47	1946	1785	1854	1785
query48	382	310	298	298
query49	790	456	459	456
query50	861	417	412	412
query51	6867	6975	6925	6925
query52	105	89	89	89
query53	257	186	179	179
query54	584	474	474	474
query55	76	72	78	72
query56	298	267	274	267
query57	1238	1092	1091	1091
query58	224	236	234	234
query59	2988	2912	2775	2775
query60	305	284	285	284
query61	128	128	123	123
query62	765	636	656	636
query63	214	185	182	182
query64	2887	786	781	781
query65	3243	3103	3156	3103
query66	688	357	341	341
query67	15352	15027	15119	15027
query68	3151	595	600	595
query69	490	284	274	274
query70	1102	1119	1073	1073
query71	369	277	279	277
query72	6342	3991	3950	3950
query73	750	327	335	327
query74	9199	8928	8851	8851
query75	3338	2655	2703	2655
query76	1443	967	975	967
query77	573	327	336	327
query78	10819	9400	9041	9041
query79	1066	537	533	533
query80	762	521	509	509
query81	538	229	226	226
query82	275	140	142	140
query83	174	148	150	148
query84	271	73	76	73
query85	695	292	276	276
query86	343	307	289	289
query87	4660	4304	4345	4304
query88	3024	2378	2351	2351
query89	389	285	288	285
query90	1962	194	187	187
query91	123	103	100	100
query92	66	51	53	51
query93	1065	537	544	537
query94	759	293	320	293
query95	355	260	250	250
query96	591	271	261	261
query97	3207	3056	3063	3056
query98	220	201	204	201
query99	1527	1264	1249	1249
Total cold run time: 306577 ms
Total hot run time: 192141 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 6, 2024

ClickBench: Total hot run time: 31.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a89ce73c3db2c5d03cb9640491dd2e90d90e4f8, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.08	0.06
query5	0.48	0.49	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.56	0.49	0.49
query10	0.56	0.56	0.55
query11	0.17	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.58
query14	1.38	1.41	1.44
query15	0.83	0.83	0.84
query16	0.35	0.40	0.39
query17	1.00	0.97	1.04
query18	0.20	0.21	0.22
query19	1.85	1.70	1.75
query20	0.01	0.01	0.01
query21	15.39	0.87	0.86
query22	5.09	6.16	2.21
query23	18.28	1.47	1.31
query24	2.19	0.23	0.24
query25	0.15	0.08	0.08
query26	0.27	0.18	0.19
query27	0.08	0.08	0.08
query28	13.19	1.01	0.99
query29	12.62	3.31	3.29
query30	0.24	0.05	0.06
query31	2.88	0.40	0.39
query32	3.23	0.47	0.47
query33	2.98	2.93	3.03
query34	17.01	4.34	4.42
query35	4.41	4.40	4.48
query36	0.66	0.48	0.47
query37	0.19	0.15	0.17
query38	0.16	0.15	0.14
query39	0.05	0.03	0.04
query40	0.16	0.12	0.12
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.76 s
Total hot run time: 31.69 s

liaoxin01
liaoxin01 approved these changes Sep 9, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 9, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Sep 9, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 258ec3d into apache:master Sep 9, 2024
dataroaring pushed a commit that referenced this pull request Sep 9, 2024
@sollhui @dataroaring
[fix](cloud) should do check before abort transaction (#40463)
1d029ff 
When routine load task transaction is abort, it should do check before
abort transaction, otherwise, it may cause concurrent modifications to
the `routineLoadTaskInfoList`, which in the Java language may result in
elements in the list being null, leading to a loop throwing
NullPointerException during scheduling and making it impossible to
schedule routine load task to consume Kafka stream.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sollhui @doris-robot @liaoxin01 @dataroaring