Skip to content

[fix](index) fix inverted index compound file entry size int32 overflow#38891

Merged
xiaokang merged 1 commit intoapache:masterfrom
xiaokang:fix_index_compound_size_overflow
Aug 6, 2024
Merged

[fix](index) fix inverted index compound file entry size int32 overflow#38891
xiaokang merged 1 commit intoapache:masterfrom
xiaokang:fix_index_compound_size_overflow

Conversation

@xiaokang
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang commented Aug 5, 2024 •
edited
Loading

change FileInfo::filesize from int32_t to int64_t to avoid overflow

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 5, 2024
@xiaokang xiaokang added usercase Important user case type label dev/2.0.x dev/2.1.x dev/3.0.x and removed doing labels Aug 5, 2024
@xiaokang
Copy link
Copy Markdown
Contributor Author

xiaokang commented Aug 5, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993
zzzxl1993 approved these changes Aug 5, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 5, 2024

PR approved by anyone and no changes requested.

airborne12
airborne12 approved these changes Aug 5, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 5, 2024

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 5, 2024

TPC-H: Total hot run time: 41555 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39014b473b8e6f138bd99f6529544402156bef50, data reload: false

------ Round 1 ----------------------------------
q1	18420	4260	4139	4139
q2	2492	204	218	204
q3	11045	1354	1387	1354
q4	10730	811	945	811
q5	8464	2991	3000	2991
q6	224	141	138	138
q7	1062	609	612	609
q8	9444	1768	1935	1768
q9	8413	6578	6588	6578
q10	8730	3838	3807	3807
q11	417	243	251	243
q12	410	225	217	217
q13	17753	2974	2945	2945
q14	274	242	242	242
q15	529	477	498	477
q16	511	402	395	395
q17	954	926	919	919
q18	8057	7318	7190	7190
q19	1491	1219	1211	1211
q20	557	315	346	315
q21	5261	4719	4776	4719
q22	346	283	289	283
Total cold run time: 115584 ms
Total hot run time: 41555 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4025	4019	4019
q2	328	229	226	226
q3	2987	2987	2984	2984
q4	1879	1816	1873	1816
q5	5260	5218	5214	5214
q6	212	130	130	130
q7	2065	1658	1649	1649
q8	3177	3244	3267	3244
q9	8250	8217	8250	8217
q10	3762	3842	3817	3817
q11	551	456	454	454
q12	712	535	556	535
q13	13335	2940	2943	2940
q14	288	253	256	253
q15	526	485	472	472
q16	454	414	392	392
q17	1752	1707	1671	1671
q18	7855	7329	7174	7174
q19	1666	1656	1663	1656
q20	1963	1743	1754	1743
q21	5384	5275	5216	5216
q22	523	470	450	450
Total cold run time: 67006 ms
Total hot run time: 54272 ms

@xiaokang xiaokang removed the dev/2.0.x label Aug 5, 2024
@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 5, 2024

TPC-DS: Total hot run time: 170052 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39014b473b8e6f138bd99f6529544402156bef50, data reload: false

query1	926	380	372	372
query2	6499	1745	1716	1716
query3	6658	218	225	218
query4	20070	17551	17371	17371
query5	4267	507	509	507
query6	300	175	175	175
query7	4604	298	293	293
query8	260	201	199	199
query9	8489	2411	2410	2410
query10	487	283	259	259
query11	10670	10202	10164	10164
query12	134	94	84	84
query13	1604	381	362	362
query14	9241	7535	7265	7265
query15	225	162	164	162
query16	7089	433	485	433
query17	952	557	541	541
query18	1918	278	284	278
query19	187	144	143	143
query20	91	84	86	84
query21	205	109	103	103
query22	4475	4268	4250	4250
query23	33964	33247	33319	33247
query24	10165	3102	3063	3063
query25	688	379	394	379
query26	1766	151	156	151
query27	2950	284	274	274
query28	6911	1999	1976	1976
query29	1315	420	411	411
query30	284	152	149	149
query31	943	747	788	747
query32	102	54	54	54
query33	686	363	311	311
query34	917	482	477	477
query35	864	721	731	721
query36	1020	870	871	870
query37	201	84	77	77
query38	2870	2785	2766	2766
query39	882	818	802	802
query40	285	115	110	110
query41	46	44	44	44
query42	125	97	103	97
query43	458	442	430	430
query44	1178	728	738	728
query45	205	182	177	177
query46	1095	827	804	804
query47	1836	1755	1746	1746
query48	358	295	292	292
query49	1170	423	434	423
query50	904	436	431	431
query51	6761	6731	6723	6723
query52	105	90	93	90
query53	253	183	177	177
query54	651	456	494	456
query55	78	73	76	73
query56	278	254	258	254
query57	1149	1054	1047	1047
query58	273	294	271	271
query59	2580	2509	2358	2358
query60	322	267	269	267
query61	96	127	97	97
query62	949	656	674	656
query63	221	190	188	188
query64	5902	1929	1877	1877
query65	3186	3101	3081	3081
query66	1440	344	332	332
query67	15369	14823	14863	14823
query68	4403	565	578	565
query69	437	307	297	297
query70	1099	1040	1100	1040
query71	379	279	285	279
query72	7208	2655	2485	2485
query73	759	328	330	328
query74	6041	5617	5685	5617
query75	3391	2747	2756	2747
query76	2302	1234	1265	1234
query77	511	323	308	308
query78	9528	8904	8850	8850
query79	1637	532	537	532
query80	1178	507	514	507
query81	551	221	225	221
query82	1067	131	126	126
query83	240	171	173	171
query84	275	80	78	78
query85	1306	327	297	297
query86	392	309	301	301
query87	3316	3139	3167	3139
query88	2914	2410	2452	2410
query89	392	286	289	286
query90	1772	191	189	189
query91	181	99	103	99
query92	69	55	51	51
query93	1506	621	611	611
query94	875	304	294	294
query95	373	257	258	257
query96	592	282	281	281
query97	3264	3063	3102	3063
query98	215	196	200	196
query99	1650	1277	1306	1277
Total cold run time: 262975 ms
Total hot run time: 170052 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 5, 2024

ClickBench: Total hot run time: 29.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39014b473b8e6f138bd99f6529544402156bef50, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.06	0.07
query5	0.50	0.49	0.47
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.50
query10	0.57	0.57	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.62	0.60	0.59
query14	0.77	0.78	0.79
query15	0.91	0.85	0.86
query16	0.35	0.36	0.35
query17	1.01	1.03	1.00
query18	0.24	0.22	0.22
query19	1.81	1.77	1.73
query20	0.01	0.01	0.02
query21	15.41	0.72	0.65
query22	3.88	8.16	1.12
query23	17.77	1.29	1.24
query24	2.28	0.23	0.22
query25	0.20	0.08	0.08
query26	0.32	0.21	0.21
query27	0.46	0.23	0.23
query28	13.16	1.00	0.97
query29	12.61	3.26	3.24
query30	0.26	0.06	0.06
query31	2.88	0.41	0.40
query32	3.22	0.50	0.49
query33	2.91	2.97	2.95
query34	15.44	4.26	4.23
query35	4.27	4.33	4.31
query36	0.68	0.49	0.49
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 107.37 s
Total hot run time: 29.74 s

@xiaokang xiaokang merged commit fc9f833 into apache:master Aug 6, 2024
This was referenced Aug 6, 2024
Merged
Merged
xiaokang added a commit to xiaokang/doris that referenced this pull request Aug 6, 2024
@xiaokang
[fix](index) fix inverted index compound file entry size int32 overfl…
4c6b4fc 
…ow (apache#38891)

change FileInfo::filesize from int32_t to int64_t to avoid overflow
xiaokang added a commit that referenced this pull request Aug 6, 2024
@xiaokang
[fix](index) fix inverted index compound file entry size int32 overflow
21a67db 
#38891 (#38928)
dataroaring pushed a commit that referenced this pull request Aug 7, 2024
@xiaokang @dataroaring
[fix](index) fix inverted index compound file entry size int32 overfl…
16b0523 
…ow (#38891)

change FileInfo::filesize from int32_t to int64_t to avoid overflow
xiaokang added a commit that referenced this pull request Aug 8, 2024
@xiaokang
[fix](index) fix inverted index compound file entry size int32 overflow
9ff1b15 
#38891 (#38929)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xiaokang @doris-robot @airborne12 @qidaye @zzzxl1993 @dataroaring