Skip to content

[Fix](Variant) sparse columns should not be added in init segment iterator#40295

Merged
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:sparse-add
Sep 4, 2024
Merged

[Fix](Variant) sparse columns should not be added in init segment iterator#40295
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:sparse-add

Conversation

@eldenmoon
Copy link
Copy Markdown
Member

@eldenmoon eldenmoon commented Sep 3, 2024

No description provided.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Copy Markdown
Member Author

eldenmoon commented Sep 3, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 3, 2024

TPC-H: Total hot run time: 38136 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 939fabad94564d4d8c78e8e6be8bc43d0eff577b, data reload: false

------ Round 1 ----------------------------------
q1	17927	4529	4464	4464
q2	2309	181	186	181
q3	10476	1112	1086	1086
q4	10128	742	725	725
q5	7722	2822	2757	2757
q6	229	135	135	135
q7	966	617	593	593
q8	9321	2046	2062	2046
q9	7205	6531	6532	6531
q10	7005	2167	2201	2167
q11	475	238	242	238
q12	399	224	227	224
q13	18736	3064	3016	3016
q14	283	240	234	234
q15	531	474	483	474
q16	570	496	493	493
q17	992	669	653	653
q18	7252	6745	6837	6745
q19	1399	1076	1095	1076
q20	678	330	329	329
q21	4029	2959	2994	2959
q22	1101	1010	1012	1010
Total cold run time: 109733 ms
Total hot run time: 38136 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4393	4312	4335	4312
q2	370	271	274	271
q3	2889	2605	2662	2605
q4	1888	1696	1665	1665
q5	5395	5387	5397	5387
q6	220	130	130	130
q7	2103	1744	1758	1744
q8	3207	3369	3332	3332
q9	8405	8405	8360	8360
q10	3449	3132	3191	3132
q11	609	488	494	488
q12	776	626	627	626
q13	12201	3074	3090	3074
q14	299	272	285	272
q15	509	486	473	473
q16	592	543	578	543
q17	1786	1487	1483	1483
q18	7860	7443	7402	7402
q19	1657	1629	1543	1543
q20	2044	1815	1813	1813
q21	5528	5078	5233	5078
q22	1098	1017	1015	1015
Total cold run time: 67278 ms
Total hot run time: 54748 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 3, 2024

TPC-DS: Total hot run time: 187750 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 939fabad94564d4d8c78e8e6be8bc43d0eff577b, data reload: false

query1	908	382	368	368
query2	6449	2043	1983	1983
query3	6650	206	210	206
query4	33802	23257	23268	23257
query5	4184	506	486	486
query6	264	165	157	157
query7	4580	291	284	284
query8	267	208	202	202
query9	8427	2448	2449	2448
query10	448	268	258	258
query11	15840	14984	15177	14984
query12	156	101	102	101
query13	1642	388	384	384
query14	9839	7332	7284	7284
query15	265	172	177	172
query16	7352	436	449	436
query17	1572	554	548	548
query18	1969	280	283	280
query19	318	144	138	138
query20	113	108	110	108
query21	215	107	104	104
query22	4333	4143	4151	4143
query23	34030	33635	33200	33200
query24	11078	2872	2802	2802
query25	618	376	403	376
query26	1164	150	156	150
query27	2285	281	274	274
query28	7288	2059	2040	2040
query29	756	413	415	413
query30	304	166	159	159
query31	981	754	779	754
query32	101	55	57	55
query33	758	282	283	282
query34	989	478	489	478
query35	833	707	725	707
query36	1092	953	930	930
query37	160	92	93	92
query38	4015	3870	3870	3870
query39	1465	1383	1377	1377
query40	201	115	111	111
query41	49	47	46	46
query42	116	94	95	94
query43	513	485	498	485
query44	1236	746	740	740
query45	197	164	167	164
query46	1101	718	750	718
query47	1890	1781	1834	1781
query48	372	297	305	297
query49	1080	418	421	418
query50	808	403	413	403
query51	7067	6926	6715	6715
query52	98	88	86	86
query53	256	182	178	178
query54	913	457	452	452
query55	81	72	76	72
query56	283	267	278	267
query57	1231	1063	1080	1063
query58	249	247	245	245
query59	3132	2995	2960	2960
query60	284	261	267	261
query61	104	101	103	101
query62	840	654	645	645
query63	217	183	185	183
query64	4167	762	664	664
query65	3229	3165	3135	3135
query66	1376	336	329	329
query67	15774	15561	15216	15216
query68	3543	584	571	571
query69	401	274	274	274
query70	1121	1152	1084	1084
query71	338	272	276	272
query72	6386	4056	3959	3959
query73	743	331	332	331
query74	9258	8872	8872	8872
query75	3417	2687	2758	2687
query76	1860	1032	990	990
query77	519	336	327	327
query78	9742	9249	9058	9058
query79	1035	530	550	530
query80	710	589	507	507
query81	464	242	241	241
query82	241	147	143	143
query83	174	154	154	154
query84	217	79	74	74
query85	682	289	274	274
query86	300	305	285	285
query87	4397	4195	4363	4195
query88	2870	2313	2311	2311
query89	384	293	282	282
query90	1791	189	192	189
query91	123	102	105	102
query92	58	50	51	50
query93	1043	541	535	535
query94	683	286	325	286
query95	339	253	253	253
query96	585	263	262	262
query97	3187	3049	3060	3049
query98	217	198	193	193
query99	1496	1291	1303	1291
Total cold run time: 282698 ms
Total hot run time: 187750 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 3, 2024

ClickBench: Total hot run time: 31.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 939fabad94564d4d8c78e8e6be8bc43d0eff577b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.05
query3	0.22	0.05	0.05
query4	1.68	0.07	0.09
query5	0.52	0.50	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.55	0.55	0.55
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.59	0.59
query14	2.01	2.04	2.08
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	0.99	1.00	0.99
query18	0.21	0.21	0.20
query19	1.91	1.85	1.87
query20	0.01	0.01	0.01
query21	15.40	0.66	0.66
query22	4.14	7.23	1.58
query23	18.22	1.42	1.34
query24	2.18	0.23	0.22
query25	0.17	0.09	0.08
query26	0.26	0.18	0.17
query27	0.09	0.07	0.07
query28	13.21	1.01	1.00
query29	12.60	3.35	3.32
query30	0.25	0.06	0.05
query31	2.88	0.40	0.40
query32	3.25	0.48	0.47
query33	2.96	2.99	3.02
query34	16.92	4.32	4.43
query35	4.47	4.46	4.45
query36	0.65	0.49	0.47
query37	0.18	0.16	0.15
query38	0.15	0.15	0.14
query39	0.05	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.56 s
Total hot run time: 31.75 s

@wm1581066 wm1581066 added the p0_c label Sep 4, 2024
airborne12
airborne12 approved these changes Sep 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 4, 2024

PR approved by anyone and no changes requested.

amorynan
amorynan approved these changes Sep 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@amorynan amorynan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 2615a03 into apache:master Sep 4, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 4, 2024
@eldenmoon
[Fix](Variant) sparse columns should not be added in init segment ite…
76bb27d 
…rator (apache#40295)
@eldenmoon eldenmoon mentioned this pull request Sep 4, 2024
Merged
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 4, 2024
@eldenmoon
[Fix](Variant) sparse columns should not be added in init segment ite…
2a389cd 
…rator (apache#40295)
eldenmoon added a commit that referenced this pull request Sep 9, 2024
@eldenmoon
[Fix](Variant) use uinque id to access column reader (#39841) (#40269)
f69063e 
#39841
#40295
gavinchou pushed a commit that referenced this pull request Sep 12, 2024
@eldenmoon @gavinchou
[Fix](Variant) sparse columns should not be added in init segment ite…
ad49871 
…rator (#40295)
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

+1 more reviewer

@amorynan amorynan amorynan approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged p0_c reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@eldenmoon @doris-robot @airborne12 @amorynan @gavinchou @wm1581066