Skip to content

[fix] (inverted index ) Fix the incorrect index size during compaction#37232

Merged
qidaye merged 1 commit intoapache:masterfrom
csun5285:fix_idx_size
Jul 8, 2024
Merged

[fix] (inverted index ) Fix the incorrect index size during compaction#37232
qidaye merged 1 commit intoapache:masterfrom
csun5285:fix_idx_size

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented Jul 3, 2024

Proposed changes

Index compaction didn't update the total size and index size.

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented Jul 3, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 3, 2024

TPC-H: Total hot run time: 39635 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false

------ Round 1 ----------------------------------
q1	17628	4355	4303	4303
q2	2019	194	191	191
q3	10466	1187	1118	1118
q4	10195	798	666	666
q5	7486	2692	2685	2685
q6	221	139	136	136
q7	964	609	610	609
q8	9256	2101	2086	2086
q9	9032	6510	6501	6501
q10	8932	3680	3735	3680
q11	449	237	238	237
q12	487	232	225	225
q13	17760	2950	3026	2950
q14	273	223	221	221
q15	517	497	494	494
q16	504	394	376	376
q17	969	630	649	630
q18	8130	7477	7368	7368
q19	5935	1481	1409	1409
q20	694	323	330	323
q21	4903	3091	3180	3091
q22	388	346	336	336
Total cold run time: 117208 ms
Total hot run time: 39635 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4210	4196	4196
q2	364	258	281	258
q3	2977	2864	2851	2851
q4	2013	1699	1718	1699
q5	5669	5541	5493	5493
q6	224	126	128	126
q7	2249	1877	1896	1877
q8	3277	3411	3439	3411
q9	8743	8608	8867	8608
q10	4100	3903	3717	3717
q11	571	513	500	500
q12	818	640	621	621
q13	17154	3143	3202	3143
q14	309	281	289	281
q15	521	482	479	479
q16	500	440	437	437
q17	1810	1545	1498	1498
q18	8047	7894	7819	7819
q19	1837	1618	1641	1618
q20	2187	1899	1911	1899
q21	5190	4901	4829	4829
q22	642	549	562	549
Total cold run time: 73572 ms
Total hot run time: 55909 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 3, 2024

TPC-DS: Total hot run time: 170839 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false

query1	927	378	374	374
query2	6438	2481	2303	2303
query3	6637	212	224	212
query4	19218	17459	17265	17265
query5	3675	468	458	458
query6	288	175	160	160
query7	4595	300	312	300
query8	330	297	318	297
query9	8435	2375	2365	2365
query10	579	300	299	299
query11	10551	9912	10090	9912
query12	116	95	81	81
query13	1645	385	370	370
query14	9354	7769	6857	6857
query15	229	177	185	177
query16	7750	266	263	263
query17	1846	534	508	508
query18	1946	268	267	267
query19	188	146	154	146
query20	86	82	80	80
query21	215	134	125	125
query22	4267	4131	4043	4043
query23	33607	33733	33688	33688
query24	10718	2936	2932	2932
query25	607	383	367	367
query26	700	150	154	150
query27	2233	321	324	321
query28	6225	2123	2130	2123
query29	909	634	631	631
query30	254	154	162	154
query31	962	748	748	748
query32	94	52	55	52
query33	665	305	295	295
query34	891	492	487	487
query35	738	639	623	623
query36	1136	1005	968	968
query37	141	79	82	79
query38	2879	2836	2810	2810
query39	918	853	858	853
query40	207	129	127	127
query41	56	56	54	54
query42	149	94	101	94
query43	557	545	542	542
query44	1065	734	720	720
query45	206	169	161	161
query46	1099	717	736	717
query47	1844	1754	1810	1754
query48	359	297	308	297
query49	826	399	412	399
query50	763	377	374	374
query51	6952	6799	6681	6681
query52	102	91	96	91
query53	362	298	285	285
query54	862	444	435	435
query55	76	72	72	72
query56	288	270	263	263
query57	1111	1023	1072	1023
query58	249	246	250	246
query59	3265	3289	3106	3106
query60	309	281	281	281
query61	97	90	126	90
query62	610	442	436	436
query63	325	291	293	291
query64	8532	2242	1709	1709
query65	3149	3106	3098	3098
query66	745	316	326	316
query67	15282	14990	15029	14990
query68	4558	535	540	535
query69	568	411	319	319
query70	1191	1051	1170	1051
query71	380	333	270	270
query72	7096	5424	2737	2737
query73	746	324	322	322
query74	5903	5577	5517	5517
query75	3324	2647	2686	2647
query76	2236	942	927	927
query77	540	305	294	294
query78	10225	9846	9792	9792
query79	2331	514	506	506
query80	920	478	472	472
query81	579	225	218	218
query82	744	106	101	101
query83	262	180	165	165
query84	232	87	82	82
query85	1513	280	267	267
query86	471	326	327	326
query87	3255	3141	3138	3138
query88	3706	2347	2374	2347
query89	480	393	378	378
query90	1750	184	186	184
query91	125	98	103	98
query92	58	51	48	48
query93	2422	497	495	495
query94	1093	189	184	184
query95	408	317	315	315
query96	602	270	263	263
query97	3155	3023	3037	3023
query98	230	210	186	186
query99	1154	837	845	837
Total cold run time: 264696 ms
Total hot run time: 170839 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 3, 2024

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false

query1	0.05	0.03	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.52	0.49
query6	1.15	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.59	0.60
query14	0.75	0.78	0.76
query15	0.86	0.81	0.81
query16	0.35	0.36	0.38
query17	1.00	1.02	1.00
query18	0.22	0.25	0.25
query19	1.86	1.73	1.75
query20	0.01	0.00	0.00
query21	15.43	0.84	0.74
query22	4.68	7.43	1.45
query23	18.22	1.38	1.33
query24	2.09	0.25	0.22
query25	0.17	0.09	0.09
query26	0.27	0.18	0.17
query27	0.09	0.08	0.08
query28	13.25	1.02	1.00
query29	12.64	3.33	3.28
query30	0.26	0.06	0.05
query31	2.86	0.40	0.38
query32	3.28	0.46	0.47
query33	2.87	2.87	2.94
query34	17.12	4.36	4.44
query35	4.47	4.49	4.50
query36	0.65	0.49	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.16	0.15
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.9 s
Total hot run time: 30.27 s

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang approved these changes Jul 6, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 6, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 6, 2024

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Jul 8, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 7482c6f into apache:master Jul 8, 2024
This was referenced Jul 15, 2024
Merged
Merged
airborne12 pushed a commit that referenced this pull request Jul 16, 2024
@csun5285
[cherry-pick](branch-2.1) fix inverted index file size (#37836)
253f929 
## Proposed changes

pick from master #37232
pick from master #37564
airborne12 pushed a commit that referenced this pull request Jul 16, 2024
@csun5285
[cherry-pick](branch-2.0) fix inverted index file size (#37838)
2ddaab3 
## Proposed changes

pick from master #37232
@csun5285 csun5285 mentioned this pull request Jul 17, 2024
Closed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[fix] (inverted index ) Fix the incorrect index size during compaction (
18e1504 
#37232)

Index compaction didn't update the total size and index size.
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@csun5285
[cherry-pick](branch-2.0) fix inverted index file size (apache#37838)
219403e 
## Proposed changes

pick from master apache#37232
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@csun5285 csun5285 mentioned this pull request Dec 12, 2024
Merged
16 tasks
airborne12 pushed a commit that referenced this pull request Dec 17, 2024
@csun5285
[fix](inverted index) fix rowset data size when enable index compacti…
28f7573 
…on (#45350)

### What problem does this PR solve?

Problem Summary:

- After performing index compaction, the index file size is not included
in the rowset's data size.
- A similar PR #37232 has already fixed this bug in the master branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.5-merged dev/3.0.1-merged meta-change reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @doris-robot @xiaokang @qidaye @airborne12 @dataroaring