[fix](array index) Fix index build failure when all array column data is null by airborne12 · Pull Request #48231 · apache/doris

airborne12 · 2025-02-24T07:36:08Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #47846

Problem Summary:
Fix wrong adding array index for building index.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

… is null

Thearas · 2025-02-24T07:36:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2025-02-24T07:37:54Z

run buildall

… is null

doris-robot · 2025-02-24T08:17:21Z

TPC-H: Total hot run time: 32654 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 390113dc2fd84fcc0c0e06ab834adccc76dea6f1, data reload: false

------ Round 1 ----------------------------------
q1	17605	5695	5307	5307
q2	2045	297	166	166
q3	10419	1333	773	773
q4	10198	1094	529	529
q5	7558	2684	2608	2608
q6	215	181	133	133
q7	987	783	605	605
q8	9304	1466	1361	1361
q9	5085	4869	4819	4819
q10	6894	2344	1885	1885
q11	505	280	265	265
q12	353	390	222	222
q13	17789	3779	3080	3080
q14	236	224	211	211
q15	524	457	457	457
q16	612	604	588	588
q17	595	892	335	335
q18	6511	6338	6270	6270
q19	1199	1088	579	579
q20	323	333	189	189
q21	3041	2309	1970	1970
q22	365	342	302	302
Total cold run time: 102363 ms
Total hot run time: 32654 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5529	5495	5545	5495
q2	247	349	231	231
q3	2182	2753	2327	2327
q4	1419	1910	1357	1357
q5	4320	4157	4300	4157
q6	251	181	126	126
q7	2069	2006	1798	1798
q8	2983	3033	2864	2864
q9	7413	7238	7247	7238
q10	3160	3290	2783	2783
q11	631	513	486	486
q12	702	742	556	556
q13	3574	4096	3316	3316
q14	284	311	260	260
q15	545	466	458	458
q16	647	689	658	658
q17	1217	1812	1393	1393
q18	7590	7453	7348	7348
q19	890	925	1147	925
q20	1993	2079	1859	1859
q21	5582	5080	4849	4849
q22	609	576	553	553
Total cold run time: 53837 ms
Total hot run time: 51037 ms

airborne12 · 2025-02-24T08:17:31Z

run buildall

doris-robot · 2025-02-24T08:36:26Z

TPC-H: Total hot run time: 31530 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false

------ Round 1 ----------------------------------
q1	17615	5418	5110	5110
q2	2047	321	175	175
q3	10373	1288	733	733
q4	10216	1023	529	529
q5	7494	2468	2316	2316
q6	194	171	140	140
q7	908	759	592	592
q8	9278	1326	1187	1187
q9	4933	4613	4696	4613
q10	6837	2312	1892	1892
q11	487	271	265	265
q12	361	362	230	230
q13	17774	3710	3092	3092
q14	233	237	222	222
q15	503	462	459	459
q16	631	606	570	570
q17	580	876	333	333
q18	6503	6180	6123	6123
q19	1074	947	543	543
q20	304	328	187	187
q21	2800	2430	1920	1920
q22	362	328	299	299
Total cold run time: 101507 ms
Total hot run time: 31530 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5187	5126	5157	5126
q2	247	344	235	235
q3	2169	2726	2335	2335
q4	1456	1855	1384	1384
q5	4231	4149	4137	4137
q6	210	166	128	128
q7	1875	1809	1692	1692
q8	2525	2455	2485	2455
q9	6859	6847	6753	6753
q10	2910	3106	2640	2640
q11	570	503	476	476
q12	662	733	585	585
q13	3285	3703	3098	3098
q14	276	277	261	261
q15	503	446	456	446
q16	631	687	623	623
q17	1109	1582	1266	1266
q18	7073	6947	7061	6947
q19	786	847	867	847
q20	1916	1975	1807	1807
q21	5168	4857	4658	4658
q22	652	557	536	536
Total cold run time: 50300 ms
Total hot run time: 48435 ms

doris-robot · 2025-02-24T08:47:54Z

TPC-DS: Total hot run time: 184798 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false

query1	849	368	361	361
query2	5625	1908	1868	1868
query3	6793	227	214	214
query4	26067	23458	23477	23458
query5	4972	694	527	527
query6	295	203	188	188
query7	4600	530	306	306
query8	291	233	224	224
query9	8627	2749	2785	2749
query10	516	323	272	272
query11	15172	15389	14883	14883
query12	172	114	107	107
query13	1663	556	423	423
query14	10814	6495	6377	6377
query15	219	195	191	191
query16	7633	634	460	460
query17	1381	725	582	582
query18	1985	414	324	324
query19	215	189	177	177
query20	138	117	116	116
query21	218	124	103	103
query22	4224	4095	4283	4095
query23	33889	33068	32928	32928
query24	7200	2374	2438	2374
query25	547	473	400	400
query26	1231	278	156	156
query27	2298	494	350	350
query28	4132	2497	2475	2475
query29	701	561	433	433
query30	256	186	161	161
query31	946	852	806	806
query32	75	64	62	62
query33	559	360	304	304
query34	788	870	556	556
query35	805	806	763	763
query36	968	989	869	869
query37	120	93	81	81
query38	4165	4153	4256	4153
query39	1472	1405	1401	1401
query40	212	117	135	117
query41	62	54	50	50
query42	128	110	105	105
query43	517	542	503	503
query44	1340	831	830	830
query45	182	169	163	163
query46	880	1132	681	681
query47	1742	1767	1718	1718
query48	424	452	348	348
query49	794	537	444	444
query50	739	730	455	455
query51	4196	4207	4134	4134
query52	106	115	98	98
query53	235	267	189	189
query54	494	486	421	421
query55	91	90	93	90
query56	270	286	270	270
query57	1114	1145	1078	1078
query58	257	235	241	235
query59	2677	2688	2519	2519
query60	293	293	277	277
query61	121	120	118	118
query62	775	720	657	657
query63	248	199	191	191
query64	4234	1000	661	661
query65	3308	3179	3129	3129
query66	1117	400	318	318
query67	15631	15401	15413	15401
query68	7875	791	552	552
query69	462	303	281	281
query70	1212	1146	1114	1114
query71	419	309	295	295
query72	5683	3587	3672	3587
query73	712	766	374	374
query74	9349	9146	8738	8738
query75	3173	3161	2699	2699
query76	3094	1176	755	755
query77	541	372	280	280
query78	9868	10104	9331	9331
query79	2037	837	628	628
query80	603	528	454	454
query81	479	277	244	244
query82	443	134	102	102
query83	175	180	159	159
query84	244	92	73	73
query85	856	343	298	298
query86	338	304	264	264
query87	4526	4564	4415	4415
query88	3842	2450	2380	2380
query89	399	323	292	292
query90	1910	202	194	194
query91	131	138	110	110
query92	70	60	58	58
query93	1151	994	573	573
query94	671	425	284	284
query95	359	269	252	252
query96	534	582	317	317
query97	2760	2838	2742	2742
query98	236	203	207	203
query99	1344	1429	1273	1273
Total cold run time: 270304 ms
Total hot run time: 184798 ms

doris-robot · 2025-02-24T08:53:14Z

ClickBench: Total hot run time: 30.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false

query1	0.04	0.05	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.11
query5	0.42	0.42	0.40
query6	1.17	0.67	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.55
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.70	2.74	2.73
query15	0.93	0.85	0.85
query16	0.39	0.38	0.37
query17	1.05	1.01	1.01
query18	0.21	0.19	0.19
query19	1.91	1.97	1.87
query20	0.01	0.01	0.02
query21	15.36	0.91	0.54
query22	0.76	1.24	0.66
query23	14.91	1.36	0.58
query24	7.91	1.35	0.75
query25	0.46	0.25	0.10
query26	0.60	0.17	0.14
query27	0.05	0.05	0.05
query28	9.21	0.85	0.42
query29	12.62	3.95	3.29
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.56	0.47
query33	3.02	3.06	3.05
query34	15.80	5.17	4.58
query35	4.56	4.57	4.58
query36	0.69	0.51	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.66 s
Total hot run time: 30.7 s

amorynan

LGTM

github-actions · 2025-02-24T08:57:29Z

PR approved by anyone and no changes requested.

doris-robot · 2025-02-24T09:09:25Z

TeamCity be ut coverage result:
Function Coverage: 44.51% (11752/26406)
Line Coverage: 34.00% (98248/288944)
Region Coverage: 33.09% (50327/152094)
Branch Coverage: 28.74% (25305/88038)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/report/index.html

doris-robot · 2025-02-24T10:35:17Z

TeamCity be ut coverage result:
Function Coverage: 44.51% (11753/26406)
Line Coverage: 34.01% (98273/288944)
Region Coverage: 33.09% (50324/152094)
Branch Coverage: 28.74% (25303/88038)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/report/index.html
Increment Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/increment_report/index.html

eldenmoon

LGTM

github-actions · 2025-02-24T12:10:06Z

PR approved by at least one committer and no changes requested.

… is null (#48231) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #47846 Problem Summary: Fix wrong adding array index for building index.

… column data is null #48231 (#48264) Cherry-picked from #48231 Co-authored-by: airborne12 <jiangkai@selectdb.com>

…ed index #47846 (#48214) cherry pick from #47846 #48231

… is null (apache#48231) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#47846 Problem Summary: Fix wrong adding array index for building index.

[fix](array index) Fix index build failure when all array column data…

390113d

… is null

[fix](array index) Fix index build failure when all array column data…

e95d98d

… is null

amorynan approved these changes Feb 24, 2025

View reviewed changes

github-actions Bot added the reviewed label Feb 24, 2025

airborne12 mentioned this pull request Feb 24, 2025

branch-2.1: [fix](array index) Correct null bitmap writing for inverted index #47846 #48214

Merged

eldenmoon approved these changes Feb 24, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 24, 2025

airborne12 merged commit 7819adf into apache:master Feb 24, 2025

airborne12 deleted the fix-array-index branch February 24, 2025 12:10

airborne12 added the dev/3.0.x label Feb 24, 2025

github-actions Bot mentioned this pull request Feb 24, 2025

branch-3.0: [fix](array index) Fix index build failure when all array column data is null #48231 #48264

Merged

airborne12 added the p0_c label Feb 25, 2025

dataroaring pushed a commit that referenced this pull request Feb 25, 2025

branch-3.0: [fix](array index) Fix index build failure when all array…

d9926da

… column data is null #48231 (#48264) Cherry-picked from #48231 Co-authored-by: airborne12 <jiangkai@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Feb 25, 2025

yiguolei pushed a commit that referenced this pull request Feb 25, 2025

branch-2.1: [fix](array index) Correct null bitmap writing for invert…

1aa57a3

…ed index #47846 (#48214) cherry pick from #47846 #48231

yiguolei added the dev/2.1.9-merged label Feb 25, 2025

yiguolei mentioned this pull request Mar 25, 2025

2.1.9 Release Note #49443

Closed

Conversation

airborne12 commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 24, 2025

Uh oh!

airborne12 commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

airborne12 commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

amorynan left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

airborne12 commented Feb 24, 2025 •

edited

Loading