[BugFix](MultiCatalog)fix iceberg table get split fail when with date type conjuct by GoGoWen · Pull Request #30162 · apache/doris

GoGoWen · 2024-01-19T13:38:19Z

Proposed changes

issue found in 2.0.4-rc05

1, have iceberg table with partitioned by date type column "day"
2 execute select * from test_table where day = "2022-01-01" will get nothing while the data exist in iceberg table, just we transfer the DateLiteral("2022-01-01") to datetime, cause we can not get the split correctly.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

GoGoWen · 2024-01-19T13:40:58Z

run buildall

doris-robot · 2024-01-19T14:11:27Z

TPC-H: Total hot run time: 38786 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f907a5bebe93659afd08523e122b503d16f2b785, data reload: false

------ Round 1 ----------------------------------
q1	17687	5233	5399	5233
q2	2039	140	129	129
q3	10794	1137	1183	1137
q4	10416	775	825	775
q5	7724	3155	3130	3130
q6	202	119	116	116
q7	850	498	482	482
q8	9231	1926	1940	1926
q9	7286	6361	6365	6361
q10	8192	3015	3049	3015
q11	418	213	198	198
q12	357	190	192	190
q13	17992	3355	3345	3345
q14	243	220	217	217
q15	538	518	509	509
q16	444	385	378	378
q17	930	578	577	577
q18	7444	6771	6697	6697
q19	1555	1357	1356	1356
q20	598	311	294	294
q21	2778	2416	2406	2406
q22	360	317	315	315
Total cold run time: 108078 ms
Total hot run time: 38786 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5463	5257	5227	5227
q2	336	216	212	212
q3	3363	3262	3205	3205
q4	2090	2060	2048	2048
q5	6043	5920	5857	5857
q6	194	116	117	116
q7	2226	1846	1874	1846
q8	3205	3383	3364	3364
q9	9008	8911	8734	8734
q10	3928	3861	3846	3846
q11	561	461	457	457
q12	784	626	614	614
q13	16911	3170	3163	3163
q14	282	254	256	254
q15	551	514	503	503
q16	508	477	463	463
q17	1878	1830	1835	1830
q18	9303	18464	9522	9522
q19	23512	1541	1523	1523
q20	4606	1933	1953	1933
q21	14571	5347	5426	5347
q22	974	555	529	529
Total cold run time: 110297 ms
Total hot run time: 60593 ms

doris-robot · 2024-01-19T14:23:00Z

TPC-DS: Total hot run time: 176857 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f907a5bebe93659afd08523e122b503d16f2b785, data reload: false

query1	931	334	333	333
query2	6557	1913	1905	1905
query3	6701	199	194	194
query4	31588	22327	22322	22322
query5	6894	542	599	542
query6	260	160	171	160
query7	4602	262	260	260
query8	229	170	185	170
query9	8313	2520	2501	2501
query10	434	218	232	218
query11	16680	15547	15544	15544
query12	119	74	66	66
query13	1666	380	373	373
query14	10547	6969	6937	6937
query15	213	175	184	175
query16	4602	235	229	229
query17	1004	482	476	476
query18	1784	279	256	256
query19	185	143	135	135
query20	72	71	71	71
query21	188	130	126	126
query22	4819	4773	4728	4728
query23	31546	30768	30841	30768
query24	11911	2818	2726	2726
query25	571	318	312	312
query26	1419	146	144	144
query27	3196	267	266	266
query28	7227	1814	1798	1798
query29	1217	633	620	620
query30	281	137	137	137
query31	922	729	766	729
query32	71	57	54	54
query33	712	217	212	212
query34	1134	476	467	467
query35	864	796	819	796
query36	1320	1277	1211	1211
query37	93	59	65	59
query38	3353	3211	3196	3196
query39	1314	1260	1265	1260
query40	207	88	82	82
query41	38	34	34	34
query42	92	79	90	79
query43	491	474	473	473
query44	1054	672	681	672
query45	190	174	174	174
query46	1052	656	640	640
query47	1659	1543	1486	1486
query48	406	328	316	316
query49	1127	280	291	280
query50	679	305	300	300
query51	5370	5194	5156	5156
query52	87	75	81	75
query53	318	257	260	257
query54	471	287	282	282
query55	84	76	76	76
query56	179	170	164	164
query57	932	907	972	907
query58	194	164	169	164
query59	2930	2668	2664	2664
query60	213	185	186	185
query61	85	96	88	88
query62	632	359	361	359
query63	281	248	257	248
query64	4979	1783	1768	1768
query65	3323	3268	3254	3254
query66	1244	318	314	314
query67	15564	14936	14942	14936
query68	14237	533	488	488
query69	590	307	299	299
query70	2276	1537	1472	1472
query71	495	208	211	208
query72	5077	2845	2854	2845
query73	3905	322	314	314
query74	6922	6358	6359	6358
query75	5012	2330	2277	2277
query76	6242	1034	1005	1005
query77	745	230	233	230
query78	9032	8857	8713	8713
query79	2014	513	493	493
query80	746	330	321	321
query81	650	220	218	218
query82	201	89	84	84
query83	172	120	126	120
query84	277	71	72	71
query85	1303	334	347	334
query86	402	404	395	395
query87	3585	3407	3414	3407
query88	2915	2259	2249	2249
query89	440	358	350	350
query90	1981	192	180	180
query91	163	124	123	123
query92	51	45	41	41
query93	1308	420	408	408
query94	1971	167	154	154
query95	492	461	449	449
query96	596	329	317	317
query97	4246	4140	4154	4140
query98	207	185	185	185
query99	1036	700	719	700
Total cold run time: 298709 ms
Total hot run time: 176857 ms

doris-robot · 2024-01-19T14:28:25Z

ClickBench: Total hot run time: 31.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f907a5bebe93659afd08523e122b503d16f2b785, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.02
query3	0.23	0.04	0.05
query4	1.71	0.08	0.08
query5	0.53	0.52	0.53
query6	1.33	0.63	0.63
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.52	0.50
query10	0.57	0.57	0.57
query11	0.12	0.09	0.09
query12	0.11	0.09	0.08
query13	0.61	0.62	0.61
query14	0.78	0.80	0.82
query15	0.78	0.77	0.78
query16	0.38	0.36	0.38
query17	1.04	1.01	1.04
query18	0.24	0.25	0.26
query19	1.84	1.80	1.80
query20	0.01	0.01	0.01
query21	15.40	0.59	0.56
query22	2.41	1.73	1.93
query23	17.49	0.88	0.67
query24	2.36	1.67	1.74
query25	0.44	0.18	0.14
query26	0.64	0.14	0.14
query27	0.06	0.05	0.05
query28	10.30	0.77	0.76
query29	12.59	3.19	3.11
query30	0.55	0.51	0.49
query31	2.77	0.34	0.35
query32	3.43	0.49	0.48
query33	3.22	3.25	3.23
query34	15.92	4.33	4.28
query35	4.46	4.43	4.30
query36	1.10	1.08	1.08
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.08	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.53 s
Total hot run time: 31.48 s

doris-robot · 2024-01-19T14:31:31Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f907a5bebe93659afd08523e122b503d16f2b785 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s

morningman

LGTM

github-actions · 2024-01-24T06:24:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-24T06:24:04Z

PR approved by anyone and no changes requested.

morningman

LGTM

morningman · 2024-01-25T15:52:41Z

run buildall

doris-robot · 2024-01-25T17:06:17Z

TPC-H: Total hot run time: 39209 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 472e1c10c2f0653831dba96929d6505e36056964, data reload: false

------ Round 1 ----------------------------------
q1	17665	5523	5494	5494
q2	2057	138	132	132
q3	10718	1175	1177	1175
q4	10463	794	816	794
q5	7755	3186	3108	3108
q6	204	124	119	119
q7	887	492	484	484
q8	9224	1952	1959	1952
q9	7270	6436	6384	6384
q10	8218	3061	3027	3027
q11	399	210	204	204
q12	362	186	193	186
q13	18021	3379	3387	3379
q14	254	209	212	209
q15	548	514	504	504
q16	438	366	375	366
q17	951	614	552	552
q18	7656	6971	6754	6754
q19	1658	1364	1350	1350
q20	607	327	316	316
q21	2795	2423	2457	2423
q22	351	297	305	297
Total cold run time: 108501 ms
Total hot run time: 39209 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5597	5483	5222	5222
q2	323	221	213	213
q3	3318	3224	3204	3204
q4	2049	2084	2098	2084
q5	5918	5849	5943	5849
q6	198	118	114	114
q7	2295	1858	1861	1858
q8	3210	3377	3401	3377
q9	8960	8984	8887	8887
q10	3929	3792	3919	3792
q11	568	439	464	439
q12	789	611	632	611
q13	16919	3147	3126	3126
q14	287	254	269	254
q15	562	512	520	512
q16	528	474	462	462
q17	1890	1900	1850	1850
q18	9471	17675	9589	9589
q19	24857	1570	1510	1510
q20	4640	1922	1923	1922
q21	14320	5315	5353	5315
q22	996	515	545	515
Total cold run time: 111624 ms
Total hot run time: 60705 ms

doris-robot · 2024-01-25T17:18:06Z

TPC-DS: Total hot run time: 186416 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 472e1c10c2f0653831dba96929d6505e36056964, data reload: false

query1	929	333	324	324
query2	6566	1979	1931	1931
query3	6700	201	200	200
query4	32365	22134	22297	22134
query5	4459	365	353	353
query6	248	151	157	151
query7	4598	262	249	249
query8	234	178	196	178
query9	8276	2594	2584	2584
query10	431	239	234	234
query11	16782	15567	15629	15567
query12	127	71	68	68
query13	1680	371	360	360
query14	10509	6965	6929	6929
query15	213	182	191	182
query16	5805	256	251	251
query17	950	467	470	467
query18	1801	260	257	257
query19	174	130	130	130
query20	68	67	69	67
query21	187	137	132	132
query22	5051	4912	4694	4694
query23	31902	30714	30788	30714
query24	11508	2834	2787	2787
query25	534	297	310	297
query26	884	140	138	138
query27	2882	278	286	278
query28	6241	1841	1815	1815
query29	929	640	626	626
query30	274	134	136	134
query31	951	729	755	729
query32	72	54	45	45
query33	650	207	206	206
query34	989	450	448	448
query35	872	718	760	718
query36	1364	1144	1248	1144
query37	94	57	61	57
query38	3358	3237	3259	3237
query39	1317	1261	1267	1261
query40	194	90	81	81
query41	38	36	34	34
query42	84	85	87	85
query43	510	483	459	459
query44	1060	685	689	685
query45	191	177	175	175
query46	1078	645	650	645
query47	1653	1527	1534	1527
query48	388	316	307	307
query49	1090	288	290	288
query50	694	307	305	305
query51	5279	5241	5183	5183
query52	91	77	77	77
query53	312	252	252	252
query54	227	182	179	179
query55	83	77	75	75
query56	189	158	169	158
query57	992	940	932	932
query58	179	146	160	146
query59	2833	2638	2707	2638
query60	203	177	181	177
query61	81	84	82	82
query62	612	377	358	358
query63	286	253	268	253
query64	4278	1760	1757	1757
query65	3323	3258	3253	3253
query66	1202	310	316	310
query67	15689	15406	15102	15102
query68	10141	543	511	511
query69	576	305	294	294
query70	1704	1505	1490	1490
query71	10401	10194	10191	10191
query72	4884	2831	2821	2821
query73	1892	315	317	315
query74	6896	6450	6452	6450
query75	4450	2291	2243	2243
query76	6214	1028	1072	1028
query77	647	219	227	219
query78	9249	8808	8709	8709
query79	1000	493	487	487
query80	525	319	302	302
query81	471	198	198	198
query82	208	80	82	80
query83	134	114	127	114
query84	283	67	66	66
query85	1019	328	314	314
query86	395	383	377	377
query87	3504	3390	3350	3350
query88	3048	2182	2196	2182
query89	428	359	351	351
query90	1943	191	179	179
query91	150	117	128	117
query92	49	43	43	43
query93	1036	453	440	440
query94	1209	159	159	159
query95	497	459	441	441
query96	608	324	321	321
query97	4245	4115	4178	4115
query98	194	187	192	187
query99	968	725	723	723
Total cold run time: 294781 ms
Total hot run time: 186416 ms

doris-robot · 2024-01-25T17:23:32Z

ClickBench: Total hot run time: 30.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 472e1c10c2f0653831dba96929d6505e36056964, data reload: false

query1	0.03	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.04	0.05
query4	1.70	0.06	0.07
query5	0.54	0.53	0.53
query6	1.27	0.73	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.55	0.51	0.51
query10	0.57	0.55	0.56
query11	0.11	0.09	0.08
query12	0.12	0.10	0.10
query13	0.61	0.62	0.60
query14	0.77	0.80	0.81
query15	0.81	0.82	0.80
query16	0.36	0.36	0.38
query17	1.00	0.99	1.00
query18	0.23	0.26	0.23
query19	1.83	1.76	1.82
query20	0.01	0.01	0.01
query21	15.42	0.55	0.56
query22	2.60	2.16	1.37
query23	17.25	0.73	0.90
query24	2.48	1.30	1.17
query25	0.32	0.33	0.13
query26	0.56	0.13	0.12
query27	0.06	0.05	0.05
query28	10.96	0.78	0.79
query29	12.66	3.16	3.11
query30	0.56	0.50	0.53
query31	2.78	0.36	0.37
query32	3.34	0.49	0.50
query33	3.32	3.31	3.32
query34	15.93	4.34	4.29
query35	4.36	4.31	4.38
query36	1.13	1.07	1.10
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.01	0.02
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 104.98 s
Total hot run time: 30.73 s

doris-robot · 2024-01-25T17:26:36Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 472e1c10c2f0653831dba96929d6505e36056964 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

…e conjunct (#30161) #30162

… to the target column (#32923) Convert the datetime type in the predicate according to the target column. And add a testcase for #32194 related #30478 #30162

… to the target column (apache#32923) Convert the datetime type in the predicate according to the target column. And add a testcase for apache#32194 related apache#30478 apache#30162

…e conjunct (apache#30161) apache#30162

fix iceberg table get split fail when with date type conjuct

f907a5b

xiaokang added the dev/2.0.5 label Jan 22, 2024

wm1581066 requested a review from morningman January 22, 2024 10:37

wm1581066 added the usercase Important user case type label label Jan 22, 2024

morningman approved these changes Jan 24, 2024

View reviewed changes

morningman mentioned this pull request Jan 24, 2024

[BugFix](MultiCatalog) Iceberg table get split fail when use date type conjunct #30161

Merged

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 24, 2024

github-actions bot added the reviewed label Jan 24, 2024

wuwenchi approved these changes Jan 25, 2024

View reviewed changes

Merge branch 'master' into fix_iceberg_catalog_issue_with_date_conjunct

472e1c1

morningman approved these changes Jan 25, 2024

View reviewed changes

yiguolei merged commit 6e8037f into apache:master Jan 27, 2024

yiguolei pushed a commit that referenced this pull request Jan 27, 2024

fix iceberg table get split fail when with date type conjuct (#30162)

e576412

wuwenchi mentioned this pull request Jan 28, 2024

[bugfix](iceberg)fix read NULL with date partition #30478

Merged

morningman pushed a commit that referenced this pull request Feb 1, 2024

[BugFix](MultiCatalog) Iceberg table get split fail when use date typ…

0f3700d

…e conjunct (#30161) #30162

morningman added dev/2.0.5-merged and removed dev/2.0.5 labels Feb 1, 2024

wuwenchi mentioned this pull request Mar 28, 2024

[bugfix](iceberg)Convert the datetime type in the predicate according to the target column #32923

Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[BugFix](MultiCatalog) Iceberg table get split fail when use date typ…

b9b3560

…e conjunct (apache#30161) apache#30162

Conversation

GoGoWen commented Jan 19, 2024

Proposed changes

Further comments

Uh oh!

GoGoWen commented Jan 19, 2024

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 24, 2024

Uh oh!

github-actions bot commented Jan 24, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants