Skip to content

[fix](compaction) fix missing flush column during compaction#38356

Merged
gavinchou merged 1 commit intoapache:masterfrom
luwei16:luwei/fix-missing-flush-column
Jul 25, 2024
Merged

[fix](compaction) fix missing flush column during compaction#38356
gavinchou merged 1 commit intoapache:masterfrom
luwei16:luwei/fix-missing-flush-column

Conversation

@luwei16
Copy link
Copy Markdown
Contributor

@luwei16 luwei16 commented Jul 25, 2024 •
edited
Loading

#37960 introduces this problem

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@luwei16
Copy link
Copy Markdown
Contributor Author

luwei16 commented Jul 25, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@luwei16
Copy link
Copy Markdown
Contributor Author

luwei16 commented Jul 25, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

TPC-H: Total hot run time: 39630 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 903c6282d8858f886bc39855c3fa6b8447ef3474, data reload: false

------ Round 1 ----------------------------------
q1	17619	4291	4268	4268
q2	2029	192	185	185
q3	10468	1147	1119	1119
q4	10188	812	699	699
q5	7517	2704	2644	2644
q6	218	136	141	136
q7	943	596	594	594
q8	9231	2033	2065	2033
q9	8756	6522	6531	6522
q10	8810	3756	3758	3756
q11	468	232	240	232
q12	461	218	219	218
q13	18747	2983	2971	2971
q14	283	233	245	233
q15	516	468	470	468
q16	480	377	376	376
q17	961	661	783	661
q18	8002	7474	7374	7374
q19	7935	1374	1388	1374
q20	702	322	326	322
q21	4883	3173	3263	3173
q22	335	278	272	272
Total cold run time: 119552 ms
Total hot run time: 39630 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4441	4258	4206	4206
q2	366	283	269	269
q3	3075	2916	2945	2916
q4	1977	1754	1820	1754
q5	5608	5559	5469	5469
q6	222	135	140	135
q7	2256	1913	1809	1809
q8	3230	3453	3399	3399
q9	8741	8861	8844	8844
q10	4177	3743	3785	3743
q11	595	499	501	499
q12	812	641	642	641
q13	15907	3171	3231	3171
q14	318	284	289	284
q15	534	498	482	482
q16	525	433	446	433
q17	1817	1543	1496	1496
q18	8200	8113	7835	7835
q19	1729	1398	1464	1398
q20	2273	1875	1901	1875
q21	10605	4909	4615	4615
q22	606	506	503	503
Total cold run time: 78014 ms
Total hot run time: 55776 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

TPC-DS: Total hot run time: 174035 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 903c6282d8858f886bc39855c3fa6b8447ef3474, data reload: false

query1	914	373	363	363
query2	6435	1979	1930	1930
query3	6635	205	220	205
query4	28284	17424	17360	17360
query5	3694	482	473	473
query6	267	184	183	183
query7	4590	288	285	285
query8	255	188	198	188
query9	8485	2408	2375	2375
query10	442	302	272	272
query11	12727	9918	9952	9918
query12	113	85	86	85
query13	1638	378	371	371
query14	10393	7643	7623	7623
query15	214	166	176	166
query16	7129	454	448	448
query17	1137	559	521	521
query18	1917	275	276	275
query19	195	143	149	143
query20	89	80	81	80
query21	204	135	127	127
query22	4403	4092	3979	3979
query23	34121	33763	33619	33619
query24	10013	2933	2906	2906
query25	595	403	383	383
query26	703	153	150	150
query27	2253	274	278	274
query28	5962	2090	2073	2073
query29	903	632	627	627
query30	246	149	148	148
query31	983	767	766	766
query32	98	53	62	53
query33	631	334	330	330
query34	903	476	506	476
query35	856	756	740	740
query36	1147	997	990	990
query37	144	82	81	81
query38	2965	2899	2866	2866
query39	915	844	866	844
query40	215	127	135	127
query41	48	47	50	47
query42	115	103	106	103
query43	523	492	489	489
query44	1068	732	731	731
query45	209	174	177	174
query46	1086	737	720	720
query47	1829	1794	1775	1775
query48	387	290	293	290
query49	874	399	420	399
query50	774	389	387	387
query51	6767	6627	6632	6627
query52	104	90	98	90
query53	356	298	289	289
query54	806	439	439	439
query55	74	73	73	73
query56	291	264	262	262
query57	1131	1087	1059	1059
query58	241	239	291	239
query59	2893	2767	2746	2746
query60	324	275	271	271
query61	100	100	95	95
query62	799	651	645	645
query63	335	286	289	286
query64	9133	2193	1655	1655
query65	3192	3102	3104	3102
query66	829	322	336	322
query67	15516	15092	14970	14970
query68	4522	547	545	545
query69	559	397	356	356
query70	1222	1146	1182	1146
query71	387	272	272	272
query72	7135	5450	6090	5450
query73	752	324	320	320
query74	6310	5664	5651	5651
query75	3380	2660	2725	2660
query76	2137	989	904	904
query77	438	307	295	295
query78	10847	10182	9133	9133
query79	4547	526	521	521
query80	1061	466	531	466
query81	559	227	215	215
query82	742	137	133	133
query83	197	166	167	166
query84	273	89	82	82
query85	1147	329	299	299
query86	460	310	330	310
query87	3254	3097	3092	3092
query88	4454	2353	2377	2353
query89	506	397	383	383
query90	1821	192	190	190
query91	127	99	97	97
query92	65	48	52	48
query93	5247	523	511	511
query94	1166	287	295	287
query95	409	309	312	309
query96	610	277	274	274
query97	3208	3047	3025	3025
query98	223	195	188	188
query99	1510	1246	1259	1246
Total cold run time: 282892 ms
Total hot run time: 174035 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 25, 2024

ClickBench: Total hot run time: 30.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 903c6282d8858f886bc39855c3fa6b8447ef3474, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.77	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.79	0.78
query15	0.87	0.82	0.82
query16	0.36	0.36	0.37
query17	1.02	0.99	0.98
query18	0.22	0.22	0.21
query19	1.80	1.69	1.66
query20	0.02	0.01	0.01
query21	15.39	0.76	0.66
query22	4.70	6.99	1.64
query23	18.34	1.33	1.18
query24	2.12	0.23	0.23
query25	0.15	0.09	0.08
query26	0.30	0.20	0.21
query27	0.46	0.23	0.23
query28	13.30	1.02	1.00
query29	12.55	3.28	3.26
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.25	0.48	0.47
query33	2.90	2.91	2.93
query34	16.97	4.35	4.32
query35	4.38	4.40	4.44
query36	0.64	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.12	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 30.17 s

dataroaring
dataroaring approved these changes Jul 25, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 25, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit c9da17b into apache:master Jul 25, 2024
dataroaring pushed a commit that referenced this pull request Jul 27, 2024
@luwei16 @dataroaring
[fix](compaction) fix missing flush column during compaction (#38356)
8f27eac 
#37960 introduces this problem
luwei16 added a commit to luwei16/Doris that referenced this pull request Aug 4, 2024
@luwei16
[fix](compaction) fix missing flush column during compaction (apache#…
a5d9332 
…38356)

apache#37960 introduces this problem
@luwei16 luwei16 mentioned this pull request Aug 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 5, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
0f69a2a 
…ows during compaction (#37960)(#38251)(#38356) (#38835)

pick master #37960 #38251 #38356
luwei16 added a commit to luwei16/Doris that referenced this pull request Aug 13, 2024
@luwei16
[fix](compaction) fix missing flush column during compaction (apache#…
8cce17f 
…38356)

apache#37960 introduces this problem
@luwei16 luwei16 mentioned this pull request Aug 13, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 14, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
e217aab 
…ows during compaction (#37960) (#38251) (#38356)  (#39263)

pick master #37960 #38251 #38356
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
2a08d5a 
…ows during compaction (apache#37960) (apache#38251) (apache#38356)  (apache#39263)

pick master apache#37960 apache#38251 apache#38356
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@luwei16 @doris-robot @gavinchou @dataroaring