Skip to content

[bugfix](iceberg)fix read NULL with date partition#30478

Merged
yiguolei merged 4 commits intoapache:masterfrom
wuwenchi:fix_iceberg_date_type
Jan 30, 2024
Merged

[bugfix](iceberg)fix read NULL with date partition#30478
yiguolei merged 4 commits intoapache:masterfrom
wuwenchi:fix_iceberg_date_type

Conversation

@wuwenchi
Copy link
Copy Markdown
Contributor

@wuwenchi wuwenchi commented Jan 28, 2024

Proposed changes

  1. for date type: convert int data to string
  2. add test for [BugFix](MultiCatalog)fix iceberg table get split fail when with date type conjuct #30162

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@wuwenchi
Copy link
Copy Markdown
Contributor Author

wuwenchi commented Jan 28, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 28, 2024

TPC-H: Total hot run time: 37544 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 197ee0e8fac02d48ef2bda25b70c28224b63a3b0, data reload: false

------ Round 1 ----------------------------------
q1	17659	4976	4901	4901
q2	2054	137	137	137
q3	10596	957	904	904
q4	4642	749	685	685
q5	7648	2773	2833	2773
q6	182	120	120	120
q7	1105	711	697	697
q8	9262	1986	2012	1986
q9	7202	6344	6330	6330
q10	8087	2447	2416	2416
q11	418	207	215	207
q12	710	286	269	269
q13	17998	3315	3279	3279
q14	268	232	242	232
q15	524	497	484	484
q16	456	409	416	409
q17	928	508	509	508
q18	7031	6311	6149	6149
q19	1570	1388	1376	1376
q20	618	338	334	334
q21	6849	3172	3052	3052
q22	806	296	303	296
Total cold run time: 106613 ms
Total hot run time: 37544 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5239	5012	4913	4913
q2	334	232	226	226
q3	3005	2791	2634	2634
q4	1865	1611	1591	1591
q5	5364	5168	5126	5126
q6	200	115	115	115
q7	2173	1799	1813	1799
q8	3085	3169	3258	3169
q9	8472	8565	10880	8565
q10	5469	3490	3427	3427
q11	539	468	458	458
q12	764	577	609	577
q13	16958	3078	3051	3051
q14	287	250	255	250
q15	532	492	492	492
q16	495	484	484	484
q17	2021	1636	1757	1636
q18	8718	11615	8677	8677
q19	23870	1581	1521	1521
q20	4605	1906	1900	1900
q21	14725	4957	4705	4705
q22	1023	487	456	456
Total cold run time: 109743 ms
Total hot run time: 55772 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 28, 2024

TPC-DS: Total hot run time: 174191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 197ee0e8fac02d48ef2bda25b70c28224b63a3b0, data reload: false

query1	948	324	328	324
query2	6578	1913	1959	1913
query3	6709	196	192	192
query4	32895	22175	22046	22046
query5	4452	358	349	349
query6	239	154	155	154
query7	4614	266	261	261
query8	235	173	179	173
query9	8531	2283	2270	2270
query10	411	209	213	209
query11	18138	15596	15338	15338
query12	124	69	66	66
query13	1690	367	369	367
query14	9219	6610	6697	6610
query15	218	187	184	184
query16	5695	254	250	250
query17	950	468	454	454
query18	1804	253	247	247
query19	170	130	137	130
query20	74	74	68	68
query21	202	135	129	129
query22	5113	4844	4949	4844
query23	31542	31107	31141	31107
query24	14533	2813	2759	2759
query25	584	309	313	309
query26	1824	147	142	142
query27	3162	288	290	288
query28	8456	1854	1848	1848
query29	2626	627	619	619
query30	1401	144	150	144
query31	2596	717	721	717
query32	74	50	47	47
query33	697	208	212	208
query34	1120	460	459	459
query35	888	793	760	760
query36	1451	1201	1230	1201
query37	91	57	58	57
query38	3264	3174	3146	3146
query39	1326	1242	1250	1242
query40	198	89	81	81
query41	36	35	34	34
query42	81	79	82	79
query43	541	480	491	480
query44	1063	686	698	686
query45	195	177	180	177
query46	1054	660	655	655
query47	1639	1427	1457	1427
query48	394	297	308	297
query49	1140	291	277	277
query50	681	311	313	311
query51	5373	5235	5139	5139
query52	86	84	74	74
query53	321	255	265	255
query54	255	180	180	180
query55	82	74	76	74
query56	171	161	165	161
query57	1017	947	926	926
query58	179	149	158	149
query59	2622	2628	2572	2572
query60	209	181	178	178
query61	83	81	78	78
query62	682	363	342	342
query63	275	251	261	251
query64	4831	1797	1744	1744
query65	3301	3232	3223	3223
query66	1210	333	305	305
query67	14579	14094	14304	14094
query68	5179	492	504	492
query69	475	311	296	296
query70	1648	1617	1556	1556
query71	301	212	204	204
query72	4105	2816	2838	2816
query73	699	312	311	311
query74	6877	6298	6303	6298
query75	2797	2280	2292	2280
query76	3428	1033	978	978
query77	375	237	225	225
query78	9226	8710	8510	8510
query79	1992	499	493	493
query80	1299	335	306	306
query81	537	198	200	198
query82	675	81	77	77
query83	219	117	121	117
query84	278	75	71	71
query85	2336	346	325	325
query86	530	417	392	392
query87	3460	3257	3239	3239
query88	3804	2168	2157	2157
query89	416	362	349	349
query90	1953	192	183	183
query91	150	132	116	116
query92	54	43	43	43
query93	2085	448	415	415
query94	1352	159	160	159
query95	505	451	448	448
query96	634	309	318	309
query97	4251	4124	4090	4090
query98	200	197	184	184
query99	1061	674	729	674
Total cold run time: 289843 ms
Total hot run time: 174191 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 28, 2024

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 197ee0e8fac02d48ef2bda25b70c28224b63a3b0, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.05	0.05
query4	1.69	0.10	0.10
query5	0.52	0.51	0.52
query6	1.18	0.63	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.54	0.51	0.47
query10	0.54	0.55	0.57
query11	0.11	0.09	0.08
query12	0.11	0.08	0.09
query13	0.60	0.59	0.61
query14	0.78	0.79	0.79
query15	0.78	0.76	0.79
query16	0.39	0.36	0.38
query17	1.02	1.00	1.00
query18	0.21	0.27	0.23
query19	1.86	1.82	1.76
query20	0.01	0.01	0.01
query21	15.43	0.57	0.56
query22	2.38	2.27	1.46
query23	17.33	0.76	0.76
query24	2.93	0.67	0.92
query25	0.42	0.17	0.10
query26	0.57	0.14	0.13
query27	0.06	0.05	0.04
query28	12.17	0.85	0.83
query29	12.54	3.17	3.12
query30	0.66	0.56	0.58
query31	2.80	0.34	0.35
query32	3.38	0.47	0.47
query33	3.21	3.21	3.20
query34	15.87	4.24	4.33
query35	4.29	4.42	4.33
query36	1.09	1.08	1.05
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.12
query41	0.07	0.01	0.02
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.27 s
Total hot run time: 30.16 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 28, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 197ee0e8fac02d48ef2bda25b70c28224b63a3b0 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

morningman
morningman approved these changes Jan 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 29, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit d9e9926 into apache:master Jan 30, 2024
yiguolei pushed a commit that referenced this pull request Jan 30, 2024
@wuwenchi @Doris-Extras
[bugfix](iceberg)fix read NULL with date partition (#30478)
4648902 
* fix date

* fix date

* add case
@wuwenchi wuwenchi mentioned this pull request Feb 1, 2024
Merged
morningman pushed a commit that referenced this pull request Feb 1, 2024
@wuwenchi
[bugfix](iceberg)fix read NULL with date partition (#30689)
dc13a9c 
mirror #30478
@wuwenchi wuwenchi deleted the fix_iceberg_date_type branch February 19, 2024 09:12
@wuwenchi wuwenchi mentioned this pull request Mar 28, 2024
Merged
morningman pushed a commit that referenced this pull request Mar 30, 2024
@wuwenchi
[bugfix](iceberg)Convert the datetime type in the predicate according…
c19c246 
… to the target column (#32923)

Convert the datetime type in the predicate according to the target column.
And add a testcase for #32194
related #30478 #30162
yiguolei pushed a commit that referenced this pull request Apr 1, 2024
@wuwenchi @Doris-Extras
[bugfix](iceberg)Convert the datetime type in the predicate according…
32283c1 
… to the target column (#32923)

Convert the datetime type in the predicate according to the target column.
And add a testcase for #32194
related #30478 #30162
morningman pushed a commit to morningman/doris that referenced this pull request Apr 7, 2024
@wuwenchi @morningman
[bugfix](iceberg)Convert the datetime type in the predicate according…
190763e 
… to the target column (apache#32923)

Convert the datetime type in the predicate according to the target column.
And add a testcase for apache#32194
related apache#30478 apache#30162
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@wuwenchi
[bugfix](iceberg)fix read NULL with date partition (apache#30689)
a581594 
mirror apache#30478
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged p0_w reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wuwenchi @doris-robot @morningman @yiguolei @xiaokang