Skip to content

[fix](routine load) should update progress before handle transaction state transform#39311

Merged
dataroaring merged 1 commit intoapache:masterfrom
sollhui:fix_progress_cache
Aug 14, 2024
Merged

[fix](routine load) should update progress before handle transaction state transform#39311
dataroaring merged 1 commit intoapache:masterfrom
sollhui:fix_progress_cache

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented Aug 13, 2024 •
edited by dataroaring
Loading

Update progress maybe throw exception, causing offset has been persisted on edit log or meta service, but the memory data has not been updated. It will cause repeated consumption.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Aug 13, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 13, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

TPC-H: Total hot run time: 40305 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43356b6b89ce04ee0c56bb115e95752b1d63799e, data reload: false

------ Round 1 ----------------------------------
q1	17898	4473	4451	4451
q2	2355	178	181	178
q3	10491	1174	1131	1131
q4	10141	772	722	722
q5	7736	2840	2805	2805
q6	227	142	141	141
q7	981	606	589	589
q8	9318	2052	2086	2052
q9	8667	6571	6576	6571
q10	7045	2190	2207	2190
q11	450	253	250	250
q12	391	218	220	218
q13	17860	2969	2973	2969
q14	278	234	231	231
q15	528	477	500	477
q16	493	407	377	377
q17	974	642	662	642
q18	8120	7495	7448	7448
q19	5132	981	1082	981
q20	684	318	338	318
q21	5447	4555	4612	4555
q22	1143	1009	1025	1009
Total cold run time: 116359 ms
Total hot run time: 40305 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4525	4299	4311	4299
q2	381	267	260	260
q3	2860	2655	2691	2655
q4	1926	1665	1599	1599
q5	5373	5370	5343	5343
q6	222	135	134	134
q7	2048	1649	1676	1649
q8	3188	3358	3317	3317
q9	8394	8391	8429	8391
q10	3421	3173	3152	3152
q11	609	484	496	484
q12	794	628	611	611
q13	16347	2998	2971	2971
q14	299	262	283	262
q15	522	476	486	476
q16	467	410	409	409
q17	1777	1501	1486	1486
q18	7812	7479	7604	7479
q19	1677	1643	1688	1643
q20	2025	1818	1785	1785
q21	5126	5162	5126	5126
q22	1084	1017	1008	1008
Total cold run time: 70877 ms
Total hot run time: 54539 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

TPC-DS: Total hot run time: 185146 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43356b6b89ce04ee0c56bb115e95752b1d63799e, data reload: false

query1	904	390	362	362
query2	6455	1858	1849	1849
query3	6645	207	213	207
query4	29350	23206	23211	23206
query5	4190	496	500	496
query6	280	171	164	164
query7	4591	297	303	297
query8	246	204	202	202
query9	8638	2430	2409	2409
query10	435	278	264	264
query11	17757	15120	15083	15083
query12	182	102	94	94
query13	1608	359	348	348
query14	10006	7541	6908	6908
query15	223	172	166	166
query16	7750	452	466	452
query17	1556	568	521	521
query18	1865	282	273	273
query19	182	141	154	141
query20	111	103	108	103
query21	209	99	97	97
query22	4370	4109	4103	4103
query23	33888	33129	33511	33129
query24	12263	2850	2805	2805
query25	675	371	373	371
query26	1740	156	154	154
query27	3043	270	274	270
query28	7380	2043	2034	2034
query29	1092	400	389	389
query30	303	147	149	147
query31	960	756	778	756
query32	98	54	56	54
query33	752	287	272	272
query34	1001	468	462	462
query35	841	712	720	712
query36	1096	936	914	914
query37	290	79	78	78
query38	3890	3764	3850	3764
query39	1440	1372	1350	1350
query40	279	116	111	111
query41	43	41	41	41
query42	120	92	95	92
query43	492	479	466	466
query44	1142	725	734	725
query45	189	165	163	163
query46	1105	713	754	713
query47	1838	1764	1770	1764
query48	367	297	300	297
query49	1189	416	437	416
query50	790	397	397	397
query51	6797	6787	6741	6741
query52	99	92	88	88
query53	253	187	180	180
query54	1030	448	434	434
query55	76	75	73	73
query56	259	243	244	243
query57	1141	1069	1084	1069
query58	240	218	232	218
query59	2829	2731	2605	2605
query60	283	255	265	255
query61	96	98	93	93
query62	825	637	657	637
query63	207	177	179	177
query64	10538	2313	1730	1730
query65	3173	3146	3154	3146
query66	1330	349	337	337
query67	15514	14956	14907	14907
query68	4550	542	549	542
query69	390	262	270	262
query70	1113	1136	1166	1136
query71	404	282	265	265
query72	7117	2226	2024	2024
query73	744	340	322	322
query74	9058	8859	8799	8799
query75	3402	2654	2705	2654
query76	2814	988	1029	988
query77	506	312	320	312
query78	9503	9040	9007	9007
query79	2363	526	526	526
query80	1901	487	501	487
query81	576	224	230	224
query82	852	140	137	137
query83	250	155	149	149
query84	278	88	79	79
query85	2080	319	314	314
query86	486	297	312	297
query87	4361	4429	4262	4262
query88	3953	2389	2401	2389
query89	398	291	284	284
query90	1853	191	190	190
query91	118	95	94	94
query92	64	49	50	49
query93	2519	537	536	536
query94	942	281	295	281
query95	344	254	254	254
query96	606	271	267	267
query97	3244	3071	3040	3040
query98	224	195	202	195
query99	1561	1257	1321	1257
Total cold run time: 299123 ms
Total hot run time: 185146 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

ClickBench: Total hot run time: 30.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43356b6b89ce04ee0c56bb115e95752b1d63799e, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.09	0.09
query5	0.50	0.49	0.48
query6	1.13	0.73	0.74
query7	0.01	0.01	0.02
query8	0.05	0.04	0.05
query9	0.53	0.48	0.48
query10	0.54	0.54	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.58	0.60	0.58
query14	0.78	0.78	0.79
query15	0.87	0.81	0.81
query16	0.37	0.38	0.37
query17	0.97	0.95	1.00
query18	0.23	0.22	0.21
query19	1.80	1.76	1.84
query20	0.00	0.01	0.01
query21	15.40	0.76	0.66
query22	3.79	7.36	1.88
query23	18.27	1.31	1.24
query24	2.10	0.22	0.22
query25	0.14	0.07	0.08
query26	0.29	0.22	0.22
query27	0.46	0.23	0.23
query28	13.26	1.02	1.00
query29	12.60	3.36	3.30
query30	0.23	0.05	0.05
query31	2.88	0.38	0.40
query32	3.28	0.50	0.48
query33	2.99	2.96	2.94
query34	17.04	4.34	4.33
query35	4.44	4.40	4.40
query36	0.65	0.50	0.49
query37	0.20	0.15	0.16
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.25 s
Total hot run time: 30.69 s

liaoxin01
liaoxin01 approved these changes Aug 13, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 13, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 13, 2024

PR approved by anyone and no changes requested.

@sollhui sollhui changed the title [fix](routine load) should update progress before check state transform [fix](routine load) should update progress before handle transaction state transform Aug 13, 2024
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Aug 13, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

TPC-H: Total hot run time: 39591 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f3207c7e2b25cbc9fcd74791ebeb95201880fa3, data reload: false

------ Round 1 ----------------------------------
q1	17604	4285	4231	4231
q2	2023	177	171	171
q3	11869	994	1103	994
q4	10514	791	687	687
q5	7730	2764	2801	2764
q6	218	132	135	132
q7	931	588	577	577
q8	9346	2012	2019	2012
q9	8570	6522	6584	6522
q10	6999	2148	2181	2148
q11	452	244	244	244
q12	388	220	220	220
q13	17756	2966	2974	2966
q14	274	233	230	230
q15	522	479	488	479
q16	500	379	387	379
q17	954	672	640	640
q18	8014	7507	7316	7316
q19	4505	1082	1056	1056
q20	641	331	321	321
q21	5270	4497	4487	4487
q22	1129	1015	1025	1015
Total cold run time: 116209 ms
Total hot run time: 39591 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4239	4283	4239
q2	373	270	253	253
q3	2858	2575	2746	2575
q4	2026	1746	1679	1679
q5	5759	5545	5514	5514
q6	226	136	131	131
q7	2101	1829	1801	1801
q8	3286	3398	3440	3398
q9	8719	8779	8833	8779
q10	3498	3299	3190	3190
q11	589	494	516	494
q12	818	670	684	670
q13	16450	3143	3151	3143
q14	319	283	284	283
q15	525	502	487	487
q16	473	423	446	423
q17	1828	1537	1532	1532
q18	8129	7869	7802	7802
q19	3832	1580	1626	1580
q20	2158	1905	1879	1879
q21	7619	5395	5174	5174
q22	1131	1042	1074	1042
Total cold run time: 77113 ms
Total hot run time: 56068 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

TPC-DS: Total hot run time: 189450 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f3207c7e2b25cbc9fcd74791ebeb95201880fa3, data reload: false

query1	1258	864	861	861
query2	6451	1896	1854	1854
query3	10591	3756	3904	3756
query4	59119	23878	23079	23079
query5	5798	499	511	499
query6	491	163	168	163
query7	6524	291	293	291
query8	300	196	194	194
query9	9381	2435	2412	2412
query10	553	271	256	256
query11	18020	14911	15025	14911
query12	162	99	99	99
query13	1687	411	396	396
query14	13095	7796	7686	7686
query15	247	176	183	176
query16	7663	490	523	490
query17	1234	586	583	583
query18	1527	312	329	312
query19	274	147	143	143
query20	124	110	111	110
query21	206	99	97	97
query22	4482	4232	4430	4232
query23	34308	33254	33285	33254
query24	5491	2806	2774	2774
query25	522	375	369	369
query26	699	158	158	158
query27	1769	267	270	267
query28	3874	2059	2045	2045
query29	652	401	407	401
query30	235	153	148	148
query31	933	741	739	739
query32	95	51	54	51
query33	505	290	281	281
query34	862	459	462	459
query35	814	731	720	720
query36	1060	886	925	886
query37	135	79	78	78
query38	4064	3835	3808	3808
query39	1428	1378	1382	1378
query40	200	118	116	116
query41	48	43	43	43
query42	121	99	99	99
query43	507	447	469	447
query44	1127	736	728	728
query45	188	160	163	160
query46	1075	746	717	717
query47	1834	1753	1744	1744
query48	360	292	288	288
query49	761	420	425	420
query50	803	396	405	396
query51	6798	6692	6574	6574
query52	113	87	92	87
query53	255	185	183	183
query54	556	442	450	442
query55	75	76	79	76
query56	269	243	242	242
query57	1102	1030	1065	1030
query58	220	260	217	217
query59	2875	2617	2645	2617
query60	296	264	270	264
query61	104	97	100	97
query62	763	657	616	616
query63	205	180	178	178
query64	9118	2285	1762	1762
query65	3222	3151	3150	3150
query66	705	337	328	328
query67	15286	14702	14671	14671
query68	4706	548	542	542
query69	402	262	266	262
query70	1150	1054	1130	1054
query71	423	278	283	278
query72	6887	2324	2031	2031
query73	740	325	324	324
query74	9024	8915	8776	8776
query75	3389	2643	2708	2643
query76	2824	909	933	909
query77	548	310	318	310
query78	11168	8986	9303	8986
query79	2649	532	523	523
query80	1849	487	482	482
query81	563	223	222	222
query82	676	134	134	134
query83	289	156	156	156
query84	271	79	82	79
query85	1227	275	274	274
query86	393	305	290	290
query87	4320	4205	4234	4205
query88	4309	2433	2405	2405
query89	388	285	284	284
query90	1656	193	195	193
query91	123	97	96	96
query92	62	50	51	50
query93	3225	535	535	535
query94	741	257	289	257
query95	342	264	270	264
query96	604	274	273	273
query97	3214	3074	3045	3045
query98	237	207	199	199
query99	1537	1263	1261	1261
Total cold run time: 325555 ms
Total hot run time: 189450 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 13, 2024

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f3207c7e2b25cbc9fcd74791ebeb95201880fa3, data reload: false

query1	0.05	0.04	0.03
query2	0.07	0.04	0.05
query3	0.22	0.06	0.05
query4	1.67	0.08	0.09
query5	0.49	0.49	0.49
query6	1.13	0.73	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.13
query13	0.59	0.60	0.60
query14	0.76	0.77	0.79
query15	0.86	0.79	0.80
query16	0.37	0.38	0.36
query17	0.95	1.01	1.04
query18	0.22	0.21	0.21
query19	1.85	1.71	1.74
query20	0.01	0.01	0.01
query21	15.66	0.75	0.65
query22	4.01	6.43	2.51
query23	18.29	1.30	1.36
query24	2.20	0.22	0.21
query25	0.15	0.08	0.07
query26	0.30	0.22	0.21
query27	0.45	0.23	0.23
query28	13.26	1.01	1.01
query29	12.63	3.30	3.36
query30	0.23	0.05	0.05
query31	2.90	0.39	0.40
query32	3.27	0.48	0.48
query33	2.94	2.96	2.98
query34	17.16	4.35	4.37
query35	4.41	4.47	4.37
query36	0.65	0.45	0.47
query37	0.18	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.11
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 31.32 s

dataroaring
dataroaring approved these changes Aug 14, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Aug 14, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 7fe87dc into apache:master Aug 14, 2024
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@sollhui @dataroaring
[fix](routine load) should update progress before handle transaction …
30fdb4c 
…state transform (#39311)

Update progress maybe throw exception, causing offset has been persisted
on edit log or meta service, but the memory data has not been updated.
It will cause repeated consumption.
@sollhui sollhui mentioned this pull request Aug 18, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 19, 2024
@sollhui
[fix](routine load) should update progress before handle transaction …
9647885 
…state transform (#39311) (#39526)

pick (#39311)

Update progress maybe throw exception, causing offset has been persisted
on edit log or meta service, but the memory data has not been updated.
It will cause repeated consumption.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sollhui @doris-robot @liaoxin01 @dataroaring