[bugfix](iceberg)fix datetime conversion error and data path error by wuwenchi · Pull Request #35708 · apache/doris

wuwenchi · 2024-05-31T05:35:52Z

Proposed changes

Issue #31442

The unit of the seventh parameter of ZonedDateTime.of is nanosecond, so we should multiply the microsecond by 1000.
When writing to a non-partitioned iceberg table, the data path has an extra slash

doris-robot · 2024-05-31T05:35:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

wuwenchi · 2024-05-31T05:36:00Z

run buildall

wuwenchi · 2024-05-31T06:09:00Z

run buildall

github-actions · 2024-05-31T06:14:37Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-31T06:35:02Z

TPC-H: Total hot run time: 42119 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

------ Round 1 ----------------------------------
q1	17624	4322	4753	4322
q2	2037	203	206	203
q3	10419	1218	1117	1117
q4	10180	835	836	835
q5	7482	2705	2764	2705
q6	217	139	141	139
q7	980	620	617	617
q8	9222	2153	2074	2074
q9	10685	6716	6726	6716
q10	9251	3932	3909	3909
q11	443	255	241	241
q12	491	231	231	231
q13	17314	3203	3240	3203
q14	280	240	215	215
q15	515	471	493	471
q16	465	386	393	386
q17	979	686	702	686
q18	8440	7741	7801	7741
q19	4254	1570	1580	1570
q20	624	345	323	323
q21	5139	4142	4068	4068
q22	400	347	348	347
Total cold run time: 117441 ms
Total hot run time: 42119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4628	4418	4378	4378
q2	377	267	274	267
q3	3166	2943	2906	2906
q4	1867	1617	1644	1617
q5	5507	5436	5498	5436
q6	217	125	126	125
q7	2138	1851	1788	1788
q8	3208	3366	3385	3366
q9	8623	8712	8634	8634
q10	3971	3684	3771	3684
q11	597	514	502	502
q12	809	630	633	630
q13	17221	3223	3198	3198
q14	312	264	272	264
q15	516	495	497	495
q16	498	450	444	444
q17	1825	1514	1452	1452
q18	7581	7620	7394	7394
q19	1684	1574	1596	1574
q20	1975	1761	1776	1761
q21	4970	4772	4697	4697
q22	636	520	523	520
Total cold run time: 72326 ms
Total hot run time: 55132 ms

doris-robot · 2024-05-31T06:45:39Z

TPC-DS: Total hot run time: 168858 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

query1	934	393	372	372
query2	6438	2450	2320	2320
query3	6633	200	201	200
query4	19194	17587	17384	17384
query5	4102	411	429	411
query6	247	170	152	152
query7	4586	311	289	289
query8	329	279	310	279
query9	8602	2392	2372	2372
query10	461	277	256	256
query11	10650	10094	10023	10023
query12	137	92	97	92
query13	1647	402	355	355
query14	9248	6807	6803	6803
query15	238	190	185	185
query16	7452	271	265	265
query17	1301	517	521	517
query18	1917	271	268	268
query19	192	150	149	149
query20	90	87	89	87
query21	199	128	135	128
query22	4276	3884	3861	3861
query23	33562	32911	33053	32911
query24	10835	2869	3014	2869
query25	569	355	361	355
query26	703	155	153	153
query27	2262	323	318	318
query28	5804	2044	2047	2044
query29	826	609	583	583
query30	251	151	158	151
query31	969	760	746	746
query32	98	53	54	53
query33	668	266	277	266
query34	880	472	472	472
query35	726	621	601	601
query36	1066	916	924	916
query37	101	67	65	65
query38	2854	2798	2748	2748
query39	836	798	785	785
query40	194	122	122	122
query41	55	52	55	52
query42	106	99	94	94
query43	615	554	577	554
query44	1111	722	738	722
query45	194	175	174	174
query46	1060	709	732	709
query47	1852	1764	1766	1764
query48	372	299	290	290
query49	843	370	382	370
query50	764	388	371	371
query51	6777	6654	6554	6554
query52	109	90	92	90
query53	356	284	289	284
query54	900	428	441	428
query55	75	71	69	69
query56	253	236	243	236
query57	1147	1012	1047	1012
query58	251	210	204	204
query59	3356	3382	3021	3021
query60	286	248	255	248
query61	91	86	88	86
query62	604	445	469	445
query63	315	278	285	278
query64	8521	2210	1736	1736
query65	3173	3104	3148	3104
query66	814	328	324	324
query67	15338	14807	15016	14807
query68	4522	523	519	519
query69	450	271	269	269
query70	1062	1087	1101	1087
query71	372	274	272	272
query72	7277	2733	2579	2579
query73	729	332	316	316
query74	5959	5662	5633	5633
query75	3346	2639	2618	2618
query76	2326	922	919	919
query77	395	264	264	264
query78	10138	9962	9804	9804
query79	1702	517	506	506
query80	1113	442	435	435
query81	546	231	221	221
query82	945	95	88	88
query83	251	170	185	170
query84	254	93	83	83
query85	1198	298	316	298
query86	417	309	319	309
query87	3255	3083	3201	3083
query88	3193	2392	2347	2347
query89	485	396	376	376
query90	1997	193	188	188
query91	137	107	109	107
query92	59	50	49	49
query93	1324	511	497	497
query94	1215	199	194	194
query95	410	316	320	316
query96	581	277	265	265
query97	3180	3020	3068	3020
query98	244	230	222	222
query99	1179	870	852	852
Total cold run time: 261676 ms
Total hot run time: 168858 ms

doris-robot · 2024-05-31T06:51:11Z

ClickBench: Total hot run time: 30.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.68	0.08	0.09
query5	0.49	0.49	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.58	0.60
query14	0.79	0.77	0.79
query15	0.82	0.81	0.81
query16	0.37	0.37	0.36
query17	0.95	1.01	0.97
query18	0.24	0.24	0.25
query19	1.74	1.73	1.67
query20	0.01	0.01	0.01
query21	15.72	0.65	0.65
query22	4.67	6.73	2.27
query23	18.31	1.46	1.28
query24	1.29	0.43	0.23
query25	0.15	0.08	0.08
query26	0.26	0.16	0.17
query27	0.08	0.08	0.08
query28	13.33	1.00	0.99
query29	13.76	3.35	3.29
query30	0.24	0.05	0.05
query31	2.88	0.39	0.37
query32	3.28	0.48	0.47
query33	2.87	2.91	2.92
query34	17.26	4.39	4.43
query35	4.48	4.52	4.66
query36	0.65	0.47	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.15	0.17
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.61 s
Total hot run time: 30.95 s

doris-robot · 2024-05-31T06:53:49Z

TeamCity be ut coverage result:
Function Coverage: 36.29% (9232/25442)
Line Coverage: 27.64% (75713/273971)
Region Coverage: 26.85% (39205/145994)
Branch Coverage: 23.60% (19894/84286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e0a13b7ee0daebda2d0d36bf6a50fc0193b94422_e0a13b7ee0daebda2d0d36bf6a50fc0193b94422/report/index.html

morningman · 2024-05-31T13:27:31Z

run buildall

github-actions · 2024-05-31T13:33:03Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-31T14:03:08Z

TPC-H: Total hot run time: 41603 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

------ Round 1 ----------------------------------
q1	17639	4428	4210	4210
q2	2026	198	196	196
q3	10440	1258	1139	1139
q4	10194	865	908	865
q5	7457	2690	2710	2690
q6	226	136	142	136
q7	967	638	614	614
q8	9220	2121	2130	2121
q9	9360	6708	6669	6669
q10	9601	3902	3928	3902
q11	448	245	252	245
q12	422	246	231	231
q13	17262	3212	3274	3212
q14	266	206	222	206
q15	503	467	472	467
q16	486	386	395	386
q17	969	607	702	607
q18	8384	7699	7699	7699
q19	3753	1295	1444	1295
q20	652	303	308	303
q21	5231	4096	4121	4096
q22	394	335	314	314
Total cold run time: 115900 ms
Total hot run time: 41603 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4545	4438	4519	4438
q2	387	266	266	266
q3	3145	2925	2937	2925
q4	1939	1658	1609	1609
q5	5459	5535	5510	5510
q6	229	129	135	129
q7	2180	1846	1817	1817
q8	3283	3393	3408	3393
q9	8700	8649	8725	8649
q10	4173	3898	3965	3898
q11	604	494	507	494
q12	791	600	592	592
q13	16950	3200	3186	3186
q14	301	271	290	271
q15	517	487	503	487
q16	486	443	438	438
q17	1804	1513	1546	1513
q18	8710	8639	8006	8006
q19	1866	1535	1639	1535
q20	3148	1856	1874	1856
q21	8980	4854	4850	4850
q22	597	510	521	510
Total cold run time: 78794 ms
Total hot run time: 56372 ms

doris-robot · 2024-05-31T14:09:17Z

TeamCity be ut coverage result:
Function Coverage: 36.29% (9234/25447)
Line Coverage: 27.64% (75730/274011)
Region Coverage: 26.85% (39198/146001)
Branch Coverage: 23.60% (19895/84288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9ba07f34179eae32453851c872c1cb7510ee1fa4_9ba07f34179eae32453851c872c1cb7510ee1fa4/report/index.html

doris-robot · 2024-05-31T14:14:01Z

TPC-DS: Total hot run time: 170306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

query1	923	375	377	375
query2	6446	2440	2302	2302
query3	6635	212	203	203
query4	19400	17382	17268	17268
query5	4107	447	446	446
query6	245	165	157	157
query7	4583	305	294	294
query8	320	286	275	275
query9	8421	2404	2368	2368
query10	450	306	282	282
query11	10723	10317	9985	9985
query12	137	88	94	88
query13	1623	356	357	356
query14	8892	7830	7590	7590
query15	236	189	189	189
query16	8066	265	271	265
query17	1748	523	539	523
query18	2004	270	276	270
query19	205	151	154	151
query20	93	86	89	86
query21	206	136	146	136
query22	4401	3969	4182	3969
query23	33773	33042	33096	33042
query24	10724	2820	2894	2820
query25	592	354	368	354
query26	1223	167	157	157
query27	2851	323	336	323
query28	7350	2041	2045	2041
query29	882	629	644	629
query30	267	154	150	150
query31	984	769	754	754
query32	94	56	59	56
query33	800	305	277	277
query34	974	464	463	463
query35	753	623	616	616
query36	1114	935	898	898
query37	158	69	72	69
query38	2988	2763	2767	2763
query39	863	785	791	785
query40	207	125	124	124
query41	52	50	51	50
query42	122	97	97	97
query43	597	551	574	551
query44	1221	736	748	736
query45	204	167	163	163
query46	1064	734	703	703
query47	1874	1733	1770	1733
query48	379	299	290	290
query49	998	399	442	399
query50	770	385	376	376
query51	6722	6639	6656	6639
query52	108	92	87	87
query53	359	288	287	287
query54	878	460	458	458
query55	72	69	72	69
query56	287	249	263	249
query57	1127	1033	1039	1033
query58	262	244	245	244
query59	3400	3187	3139	3139
query60	301	264	270	264
query61	97	88	90	88
query62	650	447	457	447
query63	315	290	294	290
query64	8970	2243	1735	1735
query65	3178	3063	3093	3063
query66	837	348	331	331
query67	15272	14832	14748	14748
query68	4512	545	544	544
query69	452	302	303	302
query70	1155	1074	1169	1074
query71	397	291	292	291
query72	7266	2867	2704	2704
query73	769	328	328	328
query74	6103	5573	5650	5573
query75	3426	2664	2732	2664
query76	2526	881	942	881
query77	446	300	295	295
query78	10455	9860	9723	9723
query79	2262	514	516	514
query80	1900	491	483	483
query81	542	225	220	220
query82	752	109	102	102
query83	306	181	183	181
query84	264	90	91	90
query85	1921	313	311	311
query86	498	310	292	292
query87	3422	3129	3106	3106
query88	4126	2350	2348	2348
query89	476	394	383	383
query90	1831	194	191	191
query91	136	116	109	109
query92	65	51	53	51
query93	2505	520	505	505
query94	1311	208	198	198
query95	427	331	317	317
query96	596	273	265	265
query97	3204	3067	3014	3014
query98	255	223	214	214
query99	1330	865	882	865
Total cold run time: 272213 ms
Total hot run time: 170306 ms

doris-robot · 2024-05-31T14:19:32Z

ClickBench: Total hot run time: 30.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.10
query5	0.49	0.50	0.52
query6	1.11	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.76	0.78
query15	0.82	0.82	0.81
query16	0.36	0.36	0.38
query17	1.03	1.01	0.96
query18	0.23	0.24	0.25
query19	1.89	1.80	1.67
query20	0.01	0.01	0.01
query21	15.46	0.70	0.68
query22	4.70	6.74	1.96
query23	18.28	1.38	1.25
query24	2.02	0.20	0.20
query25	0.14	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.32	1.02	1.01
query29	13.30	3.26	3.30
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.46	0.47
query33	2.84	2.92	2.88
query34	17.29	4.38	4.44
query35	4.47	4.52	4.70
query36	0.65	0.45	0.46
query37	0.17	0.16	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.14	0.15
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.67 s
Total hot run time: 30.53 s

morningman

LGTM

github-actions · 2024-05-31T15:10:12Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-31T15:10:14Z

PR approved by anyone and no changes requested.

kaka11chen

LGTM

…35708) ## Proposed changes Issue #31442  1. The unit of the seventh parameter of `ZonedDateTime.of` is nanosecond, so we should multiply the microsecond by 1000. 2. When writing to a non-partitioned iceberg table, the data path has an extra slash

…pache#35708) ## Proposed changes Issue apache#31442  1. The unit of the seventh parameter of `ZonedDateTime.of` is nanosecond, so we should multiply the microsecond by 1000. 2. When writing to a non-partitioned iceberg table, the data path has an extra slash

bp: #35708

bp: apache#35708

wuwenchi changed the title ~~[bugfix](iceberg)datetime conversion error~~ [bugfix](iceberg)fix datetime conversion error and data path error May 31, 2024

wuwenchi added 3 commits May 31, 2024 21:27

fix

d82b1ec

add test

28930af

fix

9ba07f3

morningman force-pushed the fix-iceberg-ts branch from e0a13b7 to 9ba07f3 Compare May 31, 2024 13:27

morningman added dev/2.1.x usercase Important user case type label labels May 31, 2024

morningman approved these changes May 31, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 31, 2024

github-actions bot added the reviewed label May 31, 2024

kaka11chen approved these changes May 31, 2024

View reviewed changes

yiguolei merged commit 3beb92b into apache:master May 31, 2024

yiguolei removed the dev/2.1.x label May 31, 2024

yiguolei added the dev/2.1.4-merged label May 31, 2024

yiguolei mentioned this pull request Jun 1, 2024

Release Notes 2.1.4 #35752

Closed

morningman added the dev/2.0.x label Jun 3, 2024

wuwenchi mentioned this pull request Jun 4, 2024

[bugfix](iceberg)fix datetime conversion error for 2.0 #35831

Merged

morningman pushed a commit that referenced this pull request Jun 6, 2024

[bugfix](iceberg)fix datetime conversion error for 2.0 (#35831)

ceee062

bp: #35708

morningman added dev/2.0.12-merged and removed dev/2.0.x labels Jun 6, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[bugfix](iceberg)fix datetime conversion error for 2.0 (apache#35831)

9bc870f

bp: apache#35708

Conversation

wuwenchi commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

wuwenchi commented May 31, 2024

Uh oh!

wuwenchi commented May 31, 2024

Uh oh!

github-actions bot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

morningman commented May 31, 2024

Uh oh!

github-actions bot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 31, 2024

Uh oh!

github-actions bot commented May 31, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wuwenchi commented May 31, 2024 •

edited

Loading