Skip to content

[fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369)#39749

Merged
zhannngchen merged 4 commits intoapache:branch-2.0from
zhannngchen:cp-38369-2.0
Aug 22, 2024
Merged

[fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369)#39749
zhannngchen merged 4 commits intoapache:branch-2.0from
zhannngchen:cp-38369-2.0

Conversation

@zhannngchen
Copy link
Copy Markdown
Contributor

@zhannngchen zhannngchen commented Aug 22, 2024

Proposed changes

Issue Number: close #xxx

cherry-pick #38369 and #38800

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen
Copy link
Copy Markdown
Contributor Author

zhannngchen commented Aug 22, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Aug 22, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment thread be/src/olap/simple_rowid_conversion.h
Comment thread be/test/olap/segcompaction_mow_test.cpp
@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

TeamCity be ut coverage result:
Function Coverage: 37.86% (8140/21499)
Line Coverage: 29.60% (66988/226320)
Region Coverage: 29.09% (34560/118805)
Branch Coverage: 25.00% (17808/71240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/afd8d1c989e826e6c65a51bb77fa8fd552759976_afd8d1c989e826e6c65a51bb77fa8fd552759976/report/index.html

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

TPC-H: Total hot run time: 50338 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit afd8d1c989e826e6c65a51bb77fa8fd552759976, data reload: false

------ Round 1 ----------------------------------
q1	17925	4458	4396	4396
q2	2096	205	192	192
q3	10367	1927	1945	1927
q4	10375	1282	1363	1282
q5	8328	3913	3972	3913
q6	272	149	142	142
q7	2087	1678	1657	1657
q8	9360	2770	2751	2751
q9	10682	10534	10247	10247
q10	8657	3563	3619	3563
q11	454	282	278	278
q12	491	333	340	333
q13	18372	4000	4028	4000
q14	383	347	358	347
q15	556	494	503	494
q16	704	610	605	605
q17	1150	995	989	989
q18	7249	6763	7050	6763
q19	1773	1553	1601	1553
q20	560	345	328	328
q21	4529	4202	4149	4149
q22	552	429	447	429
Total cold run time: 116922 ms
Total hot run time: 50338 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4540	4390	4317	4317
q2	360	264	264	264
q3	4199	4188	4155	4155
q4	2759	2768	2759	2759
q5	7238	7139	7168	7139
q6	257	142	134	134
q7	3260	2887	2888	2887
q8	4410	4509	4548	4509
q9	16904	16754	16821	16754
q10	4250	4299	4291	4291
q11	810	720	708	708
q12	1073	888	887	887
q13	6667	3778	3725	3725
q14	484	447	456	447
q15	540	505	506	505
q16	772	714	720	714
q17	3924	3826	3838	3826
q18	8734	8751	8699	8699
q19	1776	1760	1720	1720
q20	2383	2111	2134	2111
q21	8464	8558	8590	8558
q22	1067	1029	1003	1003
Total cold run time: 84871 ms
Total hot run time: 80112 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

TPC-DS: Total hot run time: 211633 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit afd8d1c989e826e6c65a51bb77fa8fd552759976, data reload: false

query1	973	411	410	410
query2	6752	2487	2119	2119
query3	6946	229	223	223
query4	20274	18133	17998	17998
query5	19955	6661	6658	6658
query6	325	257	282	257
query7	4199	329	328	328
query8	475	455	459	455
query9	3178	2744	2680	2680
query10	470	388	364	364
query11	11381	10861	10781	10781
query12	162	109	109	109
query13	5678	706	709	706
query14	17855	13530	13925	13530
query15	395	254	244	244
query16	6530	331	304	304
query17	1647	1468	940	940
query18	2333	475	483	475
query19	230	185	185	185
query20	115	111	114	111
query21	246	152	146	146
query22	5308	5137	5136	5136
query23	33012	32211	32233	32211
query24	7728	6612	6612	6612
query25	574	480	474	474
query26	752	189	192	189
query27	2167	333	325	325
query28	6038	2376	2334	2334
query29	2876	2808	2797	2797
query30	286	210	213	210
query31	992	816	846	816
query32	101	94	91	91
query33	469	343	339	339
query34	910	516	520	516
query35	1181	1002	999	999
query36	1279	1318	1296	1296
query37	120	89	89	89
query38	3112	2967	2942	2942
query39	1478	1439	1425	1425
query40	353	146	142	142
query41	155	160	153	153
query42	112	113	110	110
query43	684	659	770	659
query44	1170	782	781	781
query45	281	271	270	270
query46	1266	1004	1016	1004
query47	1877	1831	1991	1831
query48	1033	727	738	727
query49	866	579	589	579
query50	950	726	700	700
query51	4811	4791	4711	4711
query52	123	104	118	104
query53	506	382	378	378
query54	2693	2540	2548	2540
query55	110	104	102	102
query56	311	299	289	289
query57	1283	1120	1123	1120
query58	329	304	315	304
query59	3536	3358	3367	3358
query60	295	279	275	275
query61	165	159	166	159
query62	904	542	531	531
query63	552	396	402	396
query64	2911	1726	1582	1582
query65	3649	3612	3592	3592
query66	1267	816	813	813
query67	16276	17950	16984	16984
query68	6121	688	671	671
query69	614	407	414	407
query70	1780	1636	1631	1631
query71	448	363	359	359
query72	6375	3539	3589	3539
query73	771	368	357	357
query74	6364	5942	6010	5942
query75	4886	3770	3802	3770
query76	3890	1230	1251	1230
query77	725	418	426	418
query78	12627	12289	15719	12289
query79	7060	687	676	676
query80	1887	554	559	554
query81	554	280	284	280
query82	298	125	129	125
query83	256	214	216	214
query84	291	101	100	100
query85	1048	418	422	418
query86	384	360	338	338
query87	3264	3029	3123	3029
query88	4879	2513	2525	2513
query89	401	348	339	339
query90	1780	256	257	256
query91	191	158	165	158
query92	92	85	79	79
query93	1001	565	576	565
query94	750	253	243	243
query95	1156	1130	1119	1119
query96	654	338	342	338
query97	6497	6488	6449	6449
query98	223	215	217	215
query99	2975	1086	994	994
Total cold run time: 307212 ms
Total hot run time: 211633 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

ClickBench: Total hot run time: 31.4 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit afd8d1c989e826e6c65a51bb77fa8fd552759976, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.04	0.03
query3	0.26	0.06	0.05
query4	1.78	0.07	0.07
query5	0.54	0.53	0.52
query6	1.25	0.62	0.61
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.53	0.49	0.48
query10	0.55	0.54	0.52
query11	0.14	0.10	0.10
query12	0.13	0.10	0.10
query13	0.63	0.62	0.61
query14	0.80	0.80	0.80
query15	0.80	0.77	0.77
query16	0.36	0.37	0.37
query17	1.00	1.03	1.02
query18	0.25	0.23	0.27
query19	1.92	1.87	1.87
query20	0.01	0.01	0.02
query21	15.46	0.57	0.54
query22	2.38	2.24	1.76
query23	17.27	1.01	0.96
query24	6.70	1.96	0.87
query25	0.33	0.08	0.04
query26	0.87	0.16	0.15
query27	0.06	0.04	0.05
query28	5.71	0.79	0.73
query29	12.69	2.36	2.36
query30	0.77	0.76	0.76
query31	2.83	0.40	0.38
query32	3.37	0.51	0.50
query33	3.06	3.09	3.08
query34	15.26	4.79	4.77
query35	4.88	4.85	4.84
query36	1.07	1.01	1.02
query37	0.08	0.07	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.19	0.17	0.17
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.35 s
Total hot run time: 31.4 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 22, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit afd8d1c989e826e6c65a51bb77fa8fd552759976 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@zhannngchen zhannngchen merged commit c313446 into apache:branch-2.0 Aug 22, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@zhannngchen @Baymine
[fix](merge-on-write) segcompaction should process delete bitmap if n…
e2660a0 
…ecessary (apache#38369) (apache#39749)

cherry-pick apache#38369 and apache#38800
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zhannngchen @doris-robot