Skip to content

[fix](bloom filter)Fix drop column with bloom filter#41369

Merged
qidaye merged 1 commit intoapache:masterfrom
qidaye:fix_drop_column_with_bf
Oct 11, 2024
Merged

[fix](bloom filter)Fix drop column with bloom filter#41369
qidaye merged 1 commit intoapache:masterfrom
qidaye:fix_drop_column_with_bf

Conversation

@qidaye
Copy link
Copy Markdown
Contributor

@qidaye qidaye commented Sep 26, 2024

Proposed changes

Issue Number: close #41368

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Copy Markdown
Contributor Author

qidaye commented Sep 26, 2024

run buildall

xiaokang
xiaokang approved these changes Sep 26, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 26, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 26, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 26, 2024

TPC-H: Total hot run time: 40697 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false

------ Round 1 ----------------------------------
q1	17599	7346	7232	7232
q2	2034	288	282	282
q3	12210	1065	1173	1065
q4	10569	737	787	737
q5	7734	2830	2801	2801
q6	240	149	148	148
q7	980	613	608	608
q8	9578	1930	1934	1930
q9	8170	6355	6404	6355
q10	6993	2253	2324	2253
q11	442	248	244	244
q12	416	215	221	215
q13	17792	2980	3013	2980
q14	234	215	216	215
q15	568	537	513	513
q16	654	593	573	573
q17	966	584	466	466
q18	7054	6706	6731	6706
q19	1391	1048	1048	1048
q20	484	194	192	192
q21	3941	3310	3141	3141
q22	1088	1025	993	993
Total cold run time: 111137 ms
Total hot run time: 40697 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7242	7172	7189	7172
q2	320	223	229	223
q3	3076	2885	2952	2885
q4	2100	1845	1800	1800
q5	5722	5783	5703	5703
q6	231	151	150	150
q7	2221	1859	1849	1849
q8	3374	3462	3475	3462
q9	8903	8812	8849	8812
q10	3601	3536	3520	3520
q11	588	487	493	487
q12	878	625	631	625
q13	8048	3180	3196	3180
q14	305	272	281	272
q15	588	514	534	514
q16	699	648	634	634
q17	1816	1627	1565	1565
q18	8316	7869	7481	7481
q19	1700	1528	1356	1356
q20	2117	1874	1888	1874
q21	5772	5293	5317	5293
q22	1137	1056	1028	1028
Total cold run time: 68754 ms
Total hot run time: 59885 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 26, 2024

TPC-DS: Total hot run time: 192543 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false

query1	858	391	420	391
query2	6304	2249	2065	2065
query3	8688	193	213	193
query4	33679	23597	23494	23494
query5	3411	464	460	460
query6	275	178	172	172
query7	4193	306	330	306
query8	298	238	208	208
query9	9252	2652	2677	2652
query10	458	269	271	269
query11	17975	15209	15143	15143
query12	157	95	97	95
query13	1519	426	424	424
query14	9591	7564	7662	7564
query15	256	172	180	172
query16	7916	497	490	490
query17	1636	609	588	588
query18	2002	327	339	327
query19	364	162	160	160
query20	124	112	111	111
query21	217	118	109	109
query22	4522	4520	4453	4453
query23	35568	34317	33912	33912
query24	11033	2796	2868	2796
query25	629	413	411	411
query26	1104	168	172	168
query27	2239	297	309	297
query28	7751	2420	2427	2420
query29	806	453	433	433
query30	258	153	152	152
query31	1032	809	810	809
query32	107	54	58	54
query33	771	301	299	299
query34	907	519	505	505
query35	856	734	729	729
query36	1105	981	957	957
query37	160	93	89	89
query38	3980	4033	3876	3876
query39	1513	1432	1425	1425
query40	218	98	99	98
query41	51	48	53	48
query42	115	98	96	96
query43	530	490	490	490
query44	1251	805	806	805
query45	201	166	168	166
query46	1159	736	726	726
query47	1948	1842	1851	1842
query48	477	391	378	378
query49	890	424	430	424
query50	821	416	414	414
query51	7026	7027	6944	6944
query52	103	88	97	88
query53	261	184	189	184
query54	1209	505	495	495
query55	84	79	80	79
query56	273	271	267	267
query57	1228	1108	1116	1108
query58	244	251	250	250
query59	3182	3157	2874	2874
query60	314	264	265	264
query61	102	104	102	102
query62	817	656	670	656
query63	221	192	194	192
query64	4007	657	618	618
query65	3276	3271	3221	3221
query66	1058	307	301	301
query67	15748	15487	15416	15416
query68	4704	584	557	557
query69	572	301	306	301
query70	1149	1075	1098	1075
query71	403	272	272	272
query72	7648	3990	4087	3990
query73	768	360	351	351
query74	10223	9235	9111	9111
query75	3993	2709	2694	2694
query76	3523	955	926	926
query77	695	302	289	289
query78	10497	9734	9582	9582
query79	4783	601	603	601
query80	1819	477	473	473
query81	587	242	252	242
query82	712	142	135	135
query83	322	133	136	133
query84	293	73	84	73
query85	1889	303	286	286
query86	445	305	305	305
query87	4482	4392	4371	4371
query88	4830	2409	2376	2376
query89	415	293	295	293
query90	1998	200	185	185
query91	185	141	149	141
query92	69	47	47	47
query93	3450	572	558	558
query94	870	306	307	306
query95	364	260	276	260
query96	636	284	289	284
query97	3250	3157	3098	3098
query98	227	196	194	194
query99	1535	1315	1310	1310
Total cold run time: 307664 ms
Total hot run time: 192543 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 26, 2024

ClickBench: Total hot run time: 32.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.49	0.50	0.49
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.55	0.50	0.49
query10	0.54	0.59	0.53
query11	0.13	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.60
query14	2.69	2.72	2.83
query15	0.91	0.82	0.82
query16	0.38	0.39	0.38
query17	1.01	1.02	1.06
query18	0.19	0.19	0.19
query19	1.93	1.84	1.98
query20	0.02	0.01	0.01
query21	15.49	0.59	0.59
query22	2.70	2.30	2.59
query23	17.30	0.84	0.83
query24	2.39	1.46	0.47
query25	0.21	0.11	0.14
query26	0.28	0.14	0.14
query27	0.04	0.05	0.04
query28	11.31	1.09	1.06
query29	12.60	3.24	3.24
query30	0.24	0.06	0.06
query31	2.89	0.38	0.38
query32	3.28	0.46	0.46
query33	3.01	3.00	3.00
query34	16.93	4.45	4.45
query35	4.51	4.50	4.52
query36	0.67	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.1 s
Total hot run time: 32.72 s

@qidaye qidaye force-pushed the fix_drop_column_with_bf branch from 96977a4 to c80fe25 Compare October 8, 2024 02:45
@qidaye
Copy link
Copy Markdown
Contributor Author

qidaye commented Oct 8, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 8, 2024

TPC-H: Total hot run time: 40724 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c80fe254f2a55a426a85361973807e5e337c68fc, data reload: false

------ Round 1 ----------------------------------
q1	17899	7496	7236	7236
q2	2043	279	273	273
q3	12292	1047	1116	1047
q4	10449	778	665	665
q5	7761	2853	2821	2821
q6	239	149	146	146
q7	1030	630	612	612
q8	9361	1948	2031	1948
q9	6605	6449	6353	6353
q10	6964	2288	2309	2288
q11	439	248	252	248
q12	409	217	217	217
q13	17773	2972	3005	2972
q14	237	203	214	203
q15	567	534	522	522
q16	640	574	620	574
q17	982	588	535	535
q18	7327	6604	6698	6604
q19	1383	1093	1024	1024
q20	492	216	205	205
q21	4020	3267	3426	3267
q22	1096	964	1003	964
Total cold run time: 110008 ms
Total hot run time: 40724 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7262	7242	7228	7228
q2	336	224	225	224
q3	3079	2958	2982	2958
q4	2111	1825	1855	1825
q5	5811	5795	5771	5771
q6	224	141	142	141
q7	2305	1803	1793	1793
q8	3394	3561	3503	3503
q9	8953	8941	8850	8850
q10	3600	3552	3569	3552
q11	580	494	486	486
q12	872	638	600	600
q13	9797	3182	3195	3182
q14	311	273	280	273
q15	580	525	516	516
q16	692	650	626	626
q17	1879	1605	1603	1603
q18	8338	7922	7665	7665
q19	1719	1394	1589	1394
q20	2114	1839	1868	1839
q21	5723	5538	5357	5357
q22	1123	1041	1013	1013
Total cold run time: 70803 ms
Total hot run time: 60399 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 8, 2024

TPC-DS: Total hot run time: 192771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c80fe254f2a55a426a85361973807e5e337c68fc, data reload: false

query1	881	419	396	396
query2	6263	2105	2001	2001
query3	8684	192	200	192
query4	34320	23595	23396	23396
query5	3628	460	473	460
query6	277	176	170	170
query7	4199	298	300	298
query8	285	215	212	212
query9	9179	2662	2666	2662
query10	477	295	283	283
query11	17708	15220	15409	15220
query12	154	99	98	98
query13	1578	465	458	458
query14	9621	7463	7506	7463
query15	265	173	174	173
query16	7503	487	449	449
query17	1610	615	591	591
query18	1980	320	311	311
query19	368	161	152	152
query20	125	126	122	122
query21	231	111	104	104
query22	4850	4676	4513	4513
query23	35103	34168	33969	33969
query24	11061	2797	2878	2797
query25	617	425	406	406
query26	1150	163	165	163
query27	2584	310	296	296
query28	7726	2411	2412	2411
query29	815	441	441	441
query30	262	156	155	155
query31	1057	833	807	807
query32	100	54	56	54
query33	758	298	302	298
query34	927	519	503	503
query35	941	758	748	748
query36	1099	961	946	946
query37	146	85	88	85
query38	4010	4028	4026	4026
query39	1500	1453	1437	1437
query40	259	99	96	96
query41	47	46	46	46
query42	124	96	98	96
query43	522	476	491	476
query44	1364	835	833	833
query45	211	170	165	165
query46	1170	724	723	723
query47	1995	1888	1880	1880
query48	439	351	346	346
query49	908	414	406	406
query50	819	422	415	415
query51	7154	6915	6915	6915
query52	105	90	83	83
query53	258	179	183	179
query54	1213	481	487	481
query55	82	80	80	80
query56	302	273	294	273
query57	1254	1133	1176	1133
query58	233	236	240	236
query59	3035	2948	3161	2948
query60	310	284	277	277
query61	109	105	107	105
query62	883	667	667	667
query63	219	185	183	183
query64	5019	647	625	625
query65	3320	3233	3279	3233
query66	1372	311	324	311
query67	17038	16750	15870	15870
query68	5159	552	545	545
query69	577	295	317	295
query70	1168	1071	1002	1002
query71	401	269	273	269
query72	7684	4017	3960	3960
query73	765	342	348	342
query74	10030	9036	9086	9036
query75	4385	2662	2680	2662
query76	3543	926	967	926
query77	723	299	288	288
query78	10478	9616	9540	9540
query79	2954	601	614	601
query80	2490	451	444	444
query81	580	243	241	241
query82	708	134	135	134
query83	319	144	133	133
query84	286	77	83	77
query85	1757	306	301	301
query86	463	298	311	298
query87	4557	4347	4303	4303
query88	4356	2434	2385	2385
query89	400	292	283	283
query90	2206	190	183	183
query91	156	106	114	106
query92	67	48	47	47
query93	2378	555	554	554
query94	1274	295	287	287
query95	353	252	252	252
query96	631	277	285	277
query97	3308	3204	3176	3176
query98	223	193	191	191
query99	1619	1321	1286	1286
Total cold run time: 309804 ms
Total hot run time: 192771 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Oct 8, 2024

ClickBench: Total hot run time: 32.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c80fe254f2a55a426a85361973807e5e337c68fc, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.02	0.03
query3	0.23	0.07	0.06
query4	1.66	0.10	0.09
query5	0.53	0.51	0.52
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.58	0.53	0.50
query10	0.56	0.58	0.55
query11	0.14	0.11	0.10
query12	0.14	0.10	0.10
query13	0.61	0.60	0.59
query14	2.71	2.70	2.72
query15	0.90	0.83	0.82
query16	0.39	0.39	0.41
query17	1.00	1.05	1.04
query18	0.20	0.19	0.20
query19	1.87	1.82	2.02
query20	0.02	0.01	0.01
query21	15.37	0.60	0.59
query22	3.04	2.62	2.03
query23	17.13	1.00	0.81
query24	2.83	0.47	1.06
query25	0.20	0.15	0.13
query26	0.35	0.14	0.14
query27	0.03	0.04	0.04
query28	11.22	1.09	1.06
query29	12.56	3.21	3.17
query30	0.25	0.06	0.05
query31	2.87	0.39	0.36
query32	3.29	0.48	0.46
query33	3.03	3.05	3.02
query34	16.81	4.46	4.42
query35	4.51	4.58	4.48
query36	0.68	0.51	0.50
query37	0.09	0.06	0.05
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.48 s
Total hot run time: 32.31 s

zzzxl1993
zzzxl1993 approved these changes Oct 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 1b4cf4c into apache:master Oct 11, 2024
@qidaye qidaye deleted the fix_drop_column_with_bf branch October 11, 2024 08:01
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Oct 11, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (apache#41369)
e73b585 
Issue Number: close apache#41368
@qidaye qidaye mentioned this pull request Oct 11, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Oct 11, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (apache#41369)
871d18a 
Issue Number: close apache#41368
@qidaye qidaye mentioned this pull request Oct 11, 2024
Merged
airborne12
airborne12 reviewed Oct 11, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
}
}
olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp());
}
Copy link
Copy Markdown
Member

@airborne12 airborne12 Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be this can be better

Set<String> bfCols = olapTable.getCopiedBfColumns();
if (bfCols != null) {
    Set<String> newBfCols = bfCols.stream()
                                  .filter(bfCol -> !bfCol.equalsIgnoreCase(dropColName))
                                  .collect(Collectors.toSet());
    olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp());
}

yiguolei pushed a commit that referenced this pull request Oct 12, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (#41369) (#41711)
203f00e 
bp #41369
xiaokang pushed a commit that referenced this pull request Oct 14, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (#41369) (#41710)
cbc498e
@xiaokang xiaokang added dev/2.0.16-merged-deprecated 2.0.16 is NOT and will NOT be released and removed dev/2.0.x labels Oct 14, 2024
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Nov 6, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (apache#41369)
5a7b3fa 
Issue Number: close apache#41368
@qidaye qidaye mentioned this pull request Nov 6, 2024
Merged
airborne12 pushed a commit that referenced this pull request Nov 6, 2024
@qidaye
[fix](bloom filter)Fix drop column with bloom filter (#41369) (#43378)
79b5b9d 
bp #41369
@qidaye qidaye mentioned this pull request Nov 20, 2024
Merged
16 tasks
qidaye pushed a commit that referenced this pull request Nov 22, 2024
[fix](bloom filter)Fix drop column with bloom filter index (#44361)
074bd7c 
Problem Summary:
1. When drop column with bloom filter, we modify the bloom filter column
info
2. When replay editLog, we rebuild bloom filter info by table schema.

Related PR: #41369

### Release note

Fix drop column with bloom filter index
qidaye pushed a commit to qidaye/incubator-doris that referenced this pull request Nov 22, 2024
[fix](bloom filter)Fix drop column with bloom filter index (apache#44361
bf4f3a1 
)

Problem Summary:
1. When drop column with bloom filter, we modify the bloom filter column
info
2. When replay editLog, we rebuild bloom filter info by table schema.

Related PR: apache#41369

Fix drop column with bloom filter index
qidaye pushed a commit to qidaye/incubator-doris that referenced this pull request Nov 22, 2024
[fix](bloom filter)Fix drop column with bloom filter index (apache#44361
98aaa23 
)

Problem Summary:
1. When drop column with bloom filter, we modify the bloom filter column
info
2. When replay editLog, we rebuild bloom filter info by table schema.

Related PR: apache#41369

Fix drop column with bloom filter index
qidaye pushed a commit to qidaye/incubator-doris that referenced this pull request Nov 22, 2024
[fix](bloom filter)Fix drop column with bloom filter index (apache#44361
8a120aa 
)

Problem Summary:
1. When drop column with bloom filter, we modify the bloom filter column
info
2. When replay editLog, we rebuild bloom filter info by table schema.

Related PR: apache#41369

Fix drop column with bloom filter index
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 left review comments

@xiaokang xiaokang xiaokang approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.16-merged-deprecated 2.0.16 is NOT and will NOT be released dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] drop column with bloom filter error

7 participants

@qidaye @doris-robot @xiaokang @airborne12 @zzzxl1993 @yiguolei @dataroaring