Skip to content

[opt](inverted index) Optimization of the initialization process in topn#37429

Merged
airborne12 merged 1 commit intoapache:masterfrom
zzzxl1993:202407081121
Jul 15, 2024
Merged

[opt](inverted index) Optimization of the initialization process in topn#37429
airborne12 merged 1 commit intoapache:masterfrom
zzzxl1993:202407081121

Conversation

@zzzxl1993
Copy link
Copy Markdown
Contributor

@zzzxl1993 zzzxl1993 commented Jul 8, 2024 •
edited
Loading

Proposed changes

  1. reduce the performance cost of initialization

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang reviewed Jul 8, 2024
View reviewed changes
Comment thread be/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated
nrows_read_limit = std::min(static_cast<uint32_t>(_opts.topn_limit), nrows_read_limit);
}

RETURN_IF_ERROR(_init_current_block(block, _current_return_columns));
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nrows_read_limit is not passed to _init_current_block()

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang approved these changes Jul 15, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang
Copy link
Copy Markdown
Contributor

xiaokang commented Jul 15, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2024

PR approved by anyone and no changes requested.

@xiaokang xiaokang added the doing label Jul 15, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 15, 2024

TPC-H: Total hot run time: 39811 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd9682747e50dd471133dd595a822b57b20f76f9, data reload: false

------ Round 1 ----------------------------------
q1	17663	4318	4274	4274
q2	2011	190	181	181
q3	10495	1178	1168	1168
q4	10202	763	773	763
q5	7529	2679	2600	2600
q6	223	140	140	140
q7	945	597	599	597
q8	9231	2054	2078	2054
q9	8760	6559	6540	6540
q10	8829	3847	3825	3825
q11	448	235	236	235
q12	455	221	227	221
q13	17917	2980	2952	2952
q14	290	237	243	237
q15	520	484	495	484
q16	499	379	379	379
q17	964	601	666	601
q18	7941	7443	7467	7443
q19	3732	1422	1462	1422
q20	689	318	329	318
q21	4975	3281	3099	3099
q22	346	278	290	278
Total cold run time: 114664 ms
Total hot run time: 39811 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4303	4315	4303
q2	382	273	276	273
q3	2982	2762	2905	2762
q4	1953	1689	1732	1689
q5	5558	5616	5537	5537
q6	238	141	134	134
q7	2208	1819	1883	1819
q8	3333	3440	3406	3406
q9	8751	8720	8902	8720
q10	4129	3972	3809	3809
q11	600	490	482	482
q12	809	641	624	624
q13	15843	3155	3179	3155
q14	334	292	279	279
q15	537	533	501	501
q16	460	456	442	442
q17	1784	1535	1491	1491
q18	7999	7968	7753	7753
q19	1783	1522	1511	1511
q20	2093	1957	1876	1876
q21	5133	4947	4912	4912
q22	597	505	514	505
Total cold run time: 71909 ms
Total hot run time: 55983 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 15, 2024

TPC-DS: Total hot run time: 174467 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd9682747e50dd471133dd595a822b57b20f76f9, data reload: false

query1	927	371	382	371
query2	6451	1905	1950	1905
query3	6639	213	213	213
query4	28273	17585	17285	17285
query5	3663	486	475	475
query6	269	172	157	157
query7	4584	290	298	290
query8	253	209	197	197
query9	8489	2393	2357	2357
query10	446	291	284	284
query11	11407	10161	10054	10054
query12	116	85	89	85
query13	1649	386	368	368
query14	10258	7833	7914	7833
query15	228	165	169	165
query16	7875	328	324	324
query17	1802	572	549	549
query18	2026	284	284	284
query19	201	167	159	159
query20	93	83	83	83
query21	212	127	135	127
query22	4181	4050	4022	4022
query23	33831	33661	33534	33534
query24	11042	2904	2906	2904
query25	610	418	412	412
query26	710	158	159	158
query27	2193	286	290	286
query28	6708	2014	2016	2014
query29	899	677	646	646
query30	257	159	162	159
query31	960	798	751	751
query32	100	59	64	59
query33	803	316	301	301
query34	977	510	519	510
query35	716	614	596	596
query36	1113	949	987	949
query37	149	92	90	90
query38	2973	2883	2878	2878
query39	906	869	838	838
query40	212	130	124	124
query41	52	48	47	47
query42	120	95	102	95
query43	508	479	488	479
query44	1204	739	745	739
query45	206	168	161	161
query46	1082	769	768	768
query47	1869	1783	1776	1776
query48	368	295	302	295
query49	848	517	414	414
query50	784	394	395	394
query51	6794	6808	6717	6717
query52	104	94	96	94
query53	356	294	301	294
query54	937	450	455	450
query55	76	75	81	75
query56	287	281	277	277
query57	1138	1052	1022	1022
query58	253	247	261	247
query59	2920	2688	2725	2688
query60	310	281	266	266
query61	97	93	95	93
query62	772	636	657	636
query63	315	295	292	292
query64	9139	2214	1653	1653
query65	3169	3188	3111	3111
query66	751	330	332	330
query67	15643	14917	15014	14917
query68	8160	548	550	548
query69	759	442	393	393
query70	1190	1138	1088	1088
query71	536	281	277	277
query72	8867	5938	5749	5749
query73	1480	326	321	321
query74	6258	5599	5710	5599
query75	5286	2690	2690	2690
query76	5192	949	945	945
query77	766	303	307	303
query78	10773	9139	9469	9139
query79	10436	519	527	519
query80	1041	517	475	475
query81	570	225	225	225
query82	679	135	135	135
query83	328	169	162	162
query84	276	86	88	86
query85	1316	308	297	297
query86	394	339	330	330
query87	3331	3106	3112	3106
query88	4521	2449	2481	2449
query89	528	384	375	375
query90	2054	196	227	196
query91	129	105	98	98
query92	57	52	48	48
query93	6520	504	505	504
query94	1295	211	211	211
query95	410	313	329	313
query96	625	276	275	275
query97	3206	3048	3031	3031
query98	210	200	191	191
query99	1542	1264	1263	1263
Total cold run time: 304038 ms
Total hot run time: 174467 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 15, 2024

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd9682747e50dd471133dd595a822b57b20f76f9, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.10	0.10
query5	0.51	0.49	0.50
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.53	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.77	0.76	0.80
query15	0.85	0.82	0.82
query16	0.34	0.36	0.36
query17	0.96	0.98	1.00
query18	0.23	0.21	0.22
query19	1.82	1.71	1.69
query20	0.01	0.01	0.01
query21	15.39	0.73	0.64
query22	4.28	7.03	2.51
query23	18.33	1.33	1.30
query24	2.12	0.23	0.22
query25	0.15	0.09	0.09
query26	0.28	0.22	0.20
query27	0.46	0.22	0.22
query28	13.29	1.00	0.99
query29	12.65	3.34	3.35
query30	0.26	0.06	0.05
query31	2.86	0.39	0.38
query32	3.28	0.46	0.47
query33	2.90	2.91	2.89
query34	17.00	4.35	4.38
query35	4.41	4.40	4.42
query36	0.66	0.47	0.48
query37	0.18	0.17	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.7 s
Total hot run time: 31.23 s

airborne12
airborne12 approved these changes Jul 15, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 42a311d into apache:master Jul 15, 2024
@airborne12 airborne12 removed the doing label Jul 15, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@zzzxl1993 @seawinde
[opt](inverted index) Optimization of the initialization process in t…
601b482 
…opn (apache#37429)

## Proposed changes

1. reduce the performance cost of initialization
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@zzzxl1993 @dataroaring
[opt](inverted index) Optimization of the initialization process in t…
6d2a1b0 
…opn (#37429)

## Proposed changes

1. reduce the performance cost of initialization
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jul 21, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
87f7ecb 
…opn (apache#37429)

1. reduce the performance cost of initialization
@zzzxl1993 zzzxl1993 mentioned this pull request Jul 21, 2024
Merged
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jul 23, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
5aed1a7 
…opn (apache#37429)

1. reduce the performance cost of initialization
@xiaokang xiaokang added the usercase Important user case type label label Jul 31, 2024
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 5, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
3c06a95 
…opn (apache#37429)

## Proposed changes

1. reduce the performance cost of initialization
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 5, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
69dab5c 
…opn (apache#37429)

1. reduce the performance cost of initialization
xiaokang pushed a commit that referenced this pull request Aug 6, 2024
@zzzxl1993
[opt](inverted index) mow supports index optimization #(#38180)
ff6fa33 
## Proposed changes

#37428
#37429

<!--Describe your changes.-->
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 6, 2024
@zzzxl1993
[opt](inverted index) mow supports index optimization #(apache#38180)
4ea0ce4 
apache#37428
apache#37429

<!--Describe your changes.-->
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 6, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
e32dcfa 
…opn (apache#37429)

## Proposed changes

1. reduce the performance cost of initialization
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 12, 2024
@zzzxl1993
[opt](inverted index) Optimization of the initialization process in t…
8c1ad4f 
…opn (apache#37429)

1. reduce the performance cost of initialization
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zzzxl1993 @doris-robot @xiaokang @airborne12 @dataroaring