Skip to content

[fix](cloud) fix routine load job state can not changed correctly#37347

Merged
dataroaring merged 1 commit intoapache:masterfrom
sollhui:cloud_routine_load_state_error
Jul 6, 2024
Merged

[fix](cloud) fix routine load job state can not changed correctly#37347
dataroaring merged 1 commit intoapache:masterfrom
sollhui:cloud_routine_load_state_error

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented Jul 5, 2024 •
edited
Loading

Proposed changes

introduce by #36237

If meets some error like out of range, routine load job state should change from running to pause.

When rollback transaction, the RPC will send to meta service directly, which can not change state by transaction information.

This pr sends RPC to FE first to solve this problem.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Jul 5, 2024

run buildall

liaoxin01
liaoxin01 approved these changes Jul 5, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 5, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 5, 2024

TPC-H: Total hot run time: 40750 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 771e4e9779e36d163682771273e31678e9973616, data reload: false

------ Round 1 ----------------------------------
q1	17984	4500	4455	4455
q2	2820	200	199	199
q3	10882	1151	1096	1096
q4	10551	840	764	764
q5	8539	2704	2638	2638
q6	230	142	143	142
q7	975	630	628	628
q8	9470	2044	2084	2044
q9	8586	6500	6482	6482
q10	8921	3803	3731	3731
q11	456	240	242	240
q12	405	241	244	241
q13	17759	3016	3028	3016
q14	265	226	247	226
q15	532	492	509	492
q16	529	377	384	377
q17	963	653	621	621
q18	8079	7429	7401	7401
q19	4160	1524	1378	1378
q20	667	321	329	321
q21	5005	3906	3993	3906
q22	420	352	352	352
Total cold run time: 118198 ms
Total hot run time: 40750 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4390	4285	4304	4285
q2	357	273	267	267
q3	3008	2811	2779	2779
q4	1865	1606	1617	1606
q5	5268	5297	5284	5284
q6	215	131	137	131
q7	2129	1705	1730	1705
q8	3192	3378	3343	3343
q9	8379	8333	8343	8333
q10	3893	3632	3599	3599
q11	591	521	483	483
q12	846	636	625	625
q13	17656	2993	3036	2993
q14	291	269	258	258
q15	513	485	482	482
q16	479	407	424	407
q17	1792	1477	1463	1463
q18	7700	7543	7524	7524
q19	4190	1526	1531	1526
q20	1995	1804	1764	1764
q21	4915	4766	4792	4766
q22	603	561	520	520
Total cold run time: 74267 ms
Total hot run time: 54143 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 5, 2024

TPC-DS: Total hot run time: 172215 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 771e4e9779e36d163682771273e31678e9973616, data reload: false

query1	914	377	378	377
query2	6474	2349	2274	2274
query3	6651	213	234	213
query4	28299	17357	17318	17318
query5	4261	484	497	484
query6	290	169	184	169
query7	4618	320	295	295
query8	302	283	266	266
query9	8432	2421	2403	2403
query10	612	295	278	278
query11	11688	10052	10058	10052
query12	138	87	78	78
query13	1641	371	359	359
query14	11916	7688	6767	6767
query15	237	182	186	182
query16	7282	308	294	294
query17	1851	547	515	515
query18	1195	286	273	273
query19	197	147	151	147
query20	91	81	81	81
query21	218	127	129	127
query22	4425	4216	4144	4144
query23	33748	33092	33277	33092
query24	11166	2848	2784	2784
query25	632	369	377	369
query26	1648	165	157	157
query27	2955	321	319	319
query28	7516	2094	2104	2094
query29	978	625	610	610
query30	283	145	146	145
query31	972	722	742	722
query32	96	54	54	54
query33	760	295	301	295
query34	936	499	494	494
query35	732	621	622	621
query36	1100	957	906	906
query37	148	77	78	77
query38	2893	2717	2725	2717
query39	843	811	814	811
query40	272	128	122	122
query41	54	51	52	51
query42	118	100	104	100
query43	580	561	552	552
query44	1251	755	741	741
query45	191	165	167	165
query46	1098	737	720	720
query47	1872	1801	1756	1756
query48	381	294	314	294
query49	1193	434	435	434
query50	777	398	408	398
query51	6838	6819	6852	6819
query52	107	94	98	94
query53	362	296	298	296
query54	956	455	444	444
query55	74	75	74	74
query56	310	280	269	269
query57	1137	1034	1046	1034
query58	270	240	239	239
query59	3536	3088	3191	3088
query60	311	278	275	275
query61	115	122	116	116
query62	663	445	472	445
query63	324	293	289	289
query64	10677	2197	1641	1641
query65	3201	3119	3122	3119
query66	1303	337	348	337
query67	15611	15322	14949	14949
query68	6736	563	556	556
query69	650	462	364	364
query70	1237	1143	1120	1120
query71	489	280	279	279
query72	7480	5264	5724	5264
query73	809	330	321	321
query74	6173	5472	5429	5429
query75	3973	2666	2696	2666
query76	4179	995	973	973
query77	755	310	318	310
query78	9462	9022	11792	9022
query79	2741	519	543	519
query80	1253	484	489	484
query81	558	216	218	216
query82	249	106	106	106
query83	200	166	169	166
query84	271	92	88	88
query85	994	312	312	312
query86	367	283	313	283
query87	3291	3092	3149	3092
query88	3360	2362	2342	2342
query89	472	387	407	387
query90	1939	197	194	194
query91	134	104	106	104
query92	65	51	51	51
query93	1008	511	505	505
query94	1309	215	211	211
query95	409	317	391	317
query96	583	263	263	263
query97	3176	3028	3005	3005
query98	221	206	197	197
query99	1164	845	866	845
Total cold run time: 288427 ms
Total hot run time: 172215 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 5, 2024

ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 771e4e9779e36d163682771273e31678e9973616, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.08
query5	0.50	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.49	0.48
query10	0.53	0.54	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	0.77	0.76	0.77
query15	0.86	0.82	0.82
query16	0.36	0.38	0.36
query17	1.03	1.03	0.99
query18	0.21	0.27	0.25
query19	1.81	1.75	1.73
query20	0.01	0.01	0.01
query21	15.42	0.75	0.67
query22	4.42	6.15	2.79
query23	18.24	1.39	1.17
query24	2.07	0.24	0.23
query25	0.16	0.09	0.09
query26	0.28	0.21	0.21
query27	0.46	0.23	0.23
query28	13.27	1.00	0.99
query29	12.62	3.29	3.30
query30	0.26	0.06	0.06
query31	2.88	0.39	0.39
query32	3.27	0.48	0.49
query33	2.90	2.92	3.00
query34	16.90	4.32	4.32
query35	4.45	4.36	4.40
query36	0.66	0.46	0.51
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.12
query41	0.10	0.04	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 109.76 s
Total hot run time: 31.44 s

@dataroaring dataroaring merged commit cae6429 into apache:master Jul 6, 2024
dataroaring pushed a commit that referenced this pull request Jul 6, 2024
@sollhui @dataroaring
[fix](cloud) fix routine load job state can not changed correctly (#3…
3f1414d 
…7347)

## Proposed changes

introduce by #36237

If meets some error like `out of range`, routine load job state should
change from running to pause.

When rollback transaction, the RPC will send to meta service directly,
which can not change state by transaction information.

This pr sends RPC to FE first to solve this problem.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sollhui @doris-robot @liaoxin01 @dataroaring @gavinchou