Skip to content

[fix](partial update) duplicate key occur when BE restart after conflict concurrent partial update #35739#35765

Merged
xiaokang merged 2 commits intoapache:branch-2.0from
xiaokang:pick_35739_to_apache_branch-2.0
Jun 2, 2024
Merged

[fix](partial update) duplicate key occur when BE restart after conflict concurrent partial update #35739#35765
xiaokang merged 2 commits intoapache:branch-2.0from
xiaokang:pick_35739_to_apache_branch-2.0

Conversation

@xiaokang
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang commented Jun 1, 2024

cherry pick from #35739

@xiaokang
Copy link
Copy Markdown
Contributor Author

xiaokang commented Jun 1, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xiaokang
Copy link
Copy Markdown
Contributor Author

xiaokang commented Jun 1, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

TPC-H: Total hot run time: 49713 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c296c907a4f7869f7408ae5a7c974e5daf5906fa, data reload: false

------ Round 1 ----------------------------------
q1	17766	4399	4381	4381
q2	2078	160	147	147
q3	10454	1887	1904	1887
q4	10295	1255	1333	1255
q5	8403	3945	3915	3915
q6	262	127	124	124
q7	2066	1588	1639	1588
q8	9297	2736	2723	2723
q9	10591	10318	10152	10152
q10	8663	3544	3506	3506
q11	434	249	259	249
q12	466	297	308	297
q13	18338	3936	4033	3936
q14	355	324	324	324
q15	508	464	473	464
q16	687	587	575	575
q17	1140	966	988	966
q18	7292	6937	6932	6932
q19	1710	1577	1542	1542
q20	546	306	296	296
q21	4472	4132	4072	4072
q22	484	382	406	382
Total cold run time: 116307 ms
Total hot run time: 49713 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4325	4304	4304
q2	320	227	233	227
q3	4175	4168	4181	4168
q4	2747	2760	2747	2747
q5	7170	7137	7120	7120
q6	237	119	117	117
q7	3268	2865	2867	2865
q8	4396	4479	4489	4479
q9	16877	16811	16876	16811
q10	4302	4327	4289	4289
q11	768	680	707	680
q12	1039	847	852	847
q13	7173	3747	3746	3746
q14	459	432	414	414
q15	513	455	459	455
q16	749	690	684	684
q17	3867	3891	3907	3891
q18	8865	8739	8809	8739
q19	1707	1710	1659	1659
q20	2402	2145	2113	2113
q21	8624	8565	8472	8472
q22	1061	915	915	915
Total cold run time: 85026 ms
Total hot run time: 79742 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

TeamCity be ut coverage result:
Function Coverage: 37.81% (8099/21423)
Line Coverage: 29.47% (66151/224454)
Region Coverage: 28.96% (34099/117749)
Branch Coverage: 24.81% (17500/70536)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c296c907a4f7869f7408ae5a7c974e5daf5906fa_c296c907a4f7869f7408ae5a7c974e5daf5906fa/report/index.html

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

TPC-DS: Total hot run time: 204087 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c296c907a4f7869f7408ae5a7c974e5daf5906fa, data reload: false

query1	933	394	422	394
query2	6556	2888	2664	2664
query3	6928	212	206	206
query4	21220	18095	18108	18095
query5	19748	6570	6465	6465
query6	287	213	224	213
query7	4159	303	302	302
query8	269	244	226	226
query9	3145	2708	2629	2629
query10	430	311	295	295
query11	11344	10729	10727	10727
query12	125	79	75	75
query13	5596	675	675	675
query14	18268	13417	13239	13239
query15	360	214	228	214
query16	6472	294	263	263
query17	1751	1434	873	873
query18	2315	414	416	414
query19	210	148	150	148
query20	81	79	82	79
query21	190	102	90	90
query22	5253	5085	5018	5018
query23	32636	31911	31942	31911
query24	6871	6556	6536	6536
query25	526	452	426	426
query26	521	161	162	161
query27	1824	298	296	296
query28	6205	2367	2329	2329
query29	2937	2720	2890	2720
query30	243	167	172	167
query31	917	721	747	721
query32	71	63	61	61
query33	409	269	254	254
query34	843	469	499	469
query35	1128	925	895	895
query36	1356	1206	1150	1150
query37	87	60	62	60
query38	3116	2951	2934	2934
query39	1378	1332	1321	1321
query40	220	100	96	96
query41	39	38	37	37
query42	82	81	78	78
query43	733	709	624	624
query44	1124	722	730	722
query45	239	234	230	230
query46	1221	961	974	961
query47	1795	1811	1736	1736
query48	994	715	714	714
query49	633	375	379	375
query50	876	602	643	602
query51	4768	4662	4678	4662
query52	92	83	91	83
query53	456	319	315	315
query54	2668	2553	2500	2500
query55	90	77	74	74
query56	243	218	212	212
query57	1209	1045	1087	1045
query58	218	212	211	211
query59	4159	3965	4269	3965
query60	220	224	210	210
query61	113	99	98	98
query62	811	485	476	476
query63	480	337	338	337
query64	2408	1519	1369	1369
query65	3649	3553	3574	3553
query66	767	378	387	378
query67	16117	17323	15551	15551
query68	9649	652	692	652
query69	586	379	362	362
query70	1738	1507	1436	1436
query71	414	310	312	310
query72	6706	3488	3538	3488
query73	732	323	322	322
query74	6316	5775	6026	5775
query75	5407	3720	3692	3692
query76	6212	1138	1189	1138
query77	1055	270	259	259
query78	12650	11803	12515	11803
query79	12101	634	655	634
query80	938	414	404	404
query81	486	240	236	236
query82	1427	107	98	98
query83	163	140	132	132
query84	258	71	72	71
query85	878	318	324	318
query86	325	292	300	292
query87	3226	3026	3045	3026
query88	4756	2357	2357	2357
query89	481	306	288	288
query90	1994	209	223	209
query91	177	138	138	138
query92	66	54	54	54
query93	6065	582	591	582
query94	699	211	209	209
query95	1100	1087	1072	1072
query96	653	337	332	332
query97	6509	6322	6456	6322
query98	191	178	173	173
query99	2863	1111	974	974
Total cold run time: 319953 ms
Total hot run time: 204087 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c296c907a4f7869f7408ae5a7c974e5daf5906fa, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.06	0.04
query4	1.77	0.09	0.07
query5	0.54	0.54	0.52
query6	1.23	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.53	0.49	0.48
query10	0.55	0.53	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.63	0.61	0.60
query14	0.78	0.78	0.80
query15	0.79	0.76	0.78
query16	0.38	0.36	0.35
query17	1.03	1.04	1.00
query18	0.20	0.26	0.23
query19	1.92	1.86	1.77
query20	0.02	0.01	0.01
query21	15.46	0.54	0.54
query22	2.11	2.15	1.66
query23	17.55	1.14	0.91
query24	7.19	0.95	0.91
query25	0.37	0.12	0.06
query26	0.73	0.15	0.14
query27	0.05	0.05	0.03
query28	6.11	0.73	0.75
query29	12.69	2.16	2.32
query30	0.61	0.54	0.53
query31	2.81	0.38	0.37
query32	3.40	0.50	0.50
query33	3.10	3.05	3.12
query34	15.26	4.80	4.76
query35	4.87	4.84	4.88
query36	1.10	1.01	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.78 s
Total hot run time: 30.46 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 1, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c296c907a4f7869f7408ae5a7c974e5daf5906fa with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@xiaokang xiaokang merged commit 1005311 into apache:branch-2.0 Jun 2, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@xiaokang
[fix](partial update) duplicate key occur when BE restart after confl…
60f17ce 
…ict concurrent partial update apache#35739 (apache#35765)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xiaokang @doris-robot