[fix][paimon] Fix Paimon date partition conversion failure by vinlee19 · Pull Request #54043 · apache/doris

vinlee19 · 2025-07-29T14:25:14Z

What problem does this PR solve?

Background: PR #46641 worked around Paimon date partition failures by treating date partitions as unpartitioned tables.

Root Cause: Paimon stores date partition values as integers (e.g., 20231225 for 2023-12-25), but the previous implementation failed to convert these to Doris-compatible date formats, resulting in:

org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = date literal [19737] is invalid: null

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

This reverts commit b078da1.

Thearas · 2025-07-29T14:25:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

vinlee19 · 2025-07-29T14:25:27Z

run buildall

vinlee19 · 2025-07-29T14:25:49Z

@zddr PTAL

doris-robot · 2025-07-29T14:45:50Z

TPC-H: Total hot run time: 34370 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8cd1ae637cc78b2acd86f76766bf1486f02990ef, data reload: false

------ Round 1 ----------------------------------
q1	17661	5622	5505	5505
q2	1940	303	197	197
q3	10278	1325	686	686
q4	10237	992	528	528
q5	7528	2434	2311	2311
q6	178	163	132	132
q7	938	786	598	598
q8	9297	1311	1083	1083
q9	6886	5166	5139	5139
q10	6926	2381	1989	1989
q11	471	291	275	275
q12	350	377	220	220
q13	17785	3470	2955	2955
q14	248	254	216	216
q15	520	476	473	473
q16	427	442	382	382
q17	550	857	357	357
q18	7574	7155	7143	7143
q19	1228	992	548	548
q20	311	317	207	207
q21	3393	3108	2363	2363
q22	1083	1070	1063	1063
Total cold run time: 105809 ms
Total hot run time: 34370 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5757	5687	5669	5669
q2	231	320	224	224
q3	2117	2634	2223	2223
q4	1331	1737	1316	1316
q5	4252	4380	4515	4380
q6	223	179	137	137
q7	1956	1901	1798	1798
q8	2572	2566	2428	2428
q9	7353	7524	7370	7370
q10	3182	3432	2897	2897
q11	547	514	509	509
q12	999	762	602	602
q13	3378	3754	3200	3200
q14	295	301	329	301
q15	500	460	449	449
q16	465	479	454	454
q17	1183	1485	1404	1404
q18	9061	7794	7929	7794
q19	1503	806	804	804
q20	1919	2098	2035	2035
q21	8651	4322	4231	4231
q22	1065	1046	996	996
Total cold run time: 58540 ms
Total hot run time: 51221 ms

doris-robot · 2025-07-29T14:57:50Z

TPC-DS: Total hot run time: 174767 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8cd1ae637cc78b2acd86f76766bf1486f02990ef, data reload: false

============================================
query1	1047	396	410	396
query2	6540	1925	1726	1726
query3	6753	226	225	225
query4	26945	23385	23250	23250
query5	4349	635	505	505
query6	329	238	233	233
query7	4631	518	299	299
query8	280	240	224	224
query9	8587	3209	3191	3191
query10	467	330	294	294
query11	15678	15054	14728	14728
query12	186	142	132	132
query13	1662	557	422	422
query14	9639	7717	7794	7717
query15	263	218	170	170
query16	8034	648	476	476
query17	1653	810	641	641
query18	2115	441	327	327
query19	296	215	184	184
query20	175	144	153	144
query21	222	131	109	109
query22	4043	3948	3911	3911
query23	34626	34579	34654	34579
query24	8024	2375	2403	2375
query25	578	509	441	441
query26	1066	289	163	163
query27	2656	509	352	352
query28	4198	2326	2299	2299
query29	709	602	499	499
query30	306	237	201	201
query31	894	794	735	735
query32	88	77	78	77
query33	568	414	369	369
query34	826	902	529	529
query35	818	827	784	784
query36	1013	1045	961	961
query37	126	110	95	95
query38	4110	4057	4013	4013
query39	1447	1379	1398	1379
query40	244	150	134	134
query41	68	56	56	56
query42	139	124	134	124
query43	495	504	467	467
query44	1401	896	877	877
query45	198	191	188	188
query46	943	1067	670	670
query47	1783	1845	1770	1770
query48	409	433	328	328
query49	712	507	412	412
query50	652	684	425	425
query51	5610	5610	5542	5542
query52	130	142	121	121
query53	257	295	202	202
query54	657	633	555	555
query55	92	89	89	89
query56	363	353	347	347
query57	1215	1232	1146	1146
query58	343	332	350	332
query59	2674	2584	2524	2524
query60	406	387	390	387
query61	126	141	126	126
query62	797	723	666	666
query63	257	214	208	208
query64	3838	1165	767	767
query65	4267	4110	4149	4110
query66	1096	442	337	337
query67	query68	17701	1121	953	953
query69	1012	315	292	292
query70	1436	1134	1152	1134
query71	712	340	315	315
query72	9143	2329	2459	2329
query73	3137	650	360	360
query74	8962	8858	8902	8858
query75	7543	3162	2645	2645
query76	8626	1250	808	808
query77	1166	419	395	395
query78	9624	10515	9328	9328
query79	14669	646	576	576
query80	1821	559	493	493
query81	554	259	243	243
query82	388	156	117	117
query83	354	287	277	277
query84	307	97	82	82
query85	1188	368	340	340
query86	379	330	315	315
query87	4363	4234	4181	4181
query88	4956	2274	2260	2260
query89	507	361	327	327
query90	2653	229	228	228
query91	149	142	119	119
query92	86	72	70	70
query93	6152	984	661	661
query94	1044	392	283	283
query95	393	325	319	319
query96	505	605	283	283
query97	2733	2721	2610	2610
query98	265	242	226	226
query99	1536	1420	1275	1275
Total cold run time: 309183 ms
Total hot run time: 174767 ms

doris-robot · 2025-07-29T15:03:19Z

ClickBench: Total hot run time: 33.24 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8cd1ae637cc78b2acd86f76766bf1486f02990ef, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.61	0.10	0.10
query5	0.42	0.44	0.44
query6	1.15	0.71	0.72
query7	0.03	0.02	0.01
query8	0.06	0.05	0.03
query9	0.57	0.48	0.45
query10	0.52	0.52	0.53
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.67	0.67	0.67
query14	0.97	1.09	1.10
query15	0.98	0.89	0.91
query16	0.38	0.41	0.38
query17	1.06	1.06	1.05
query18	0.21	0.21	0.20
query19	2.05	1.91	1.87
query20	0.02	0.02	0.02
query21	15.36	0.85	0.56
query22	0.79	1.33	0.72
query23	14.76	1.19	0.63
query24	7.25	1.79	0.38
query25	0.37	0.26	0.13
query26	0.69	0.16	0.14
query27	0.06	0.06	0.05
query28	8.83	0.82	0.45
query29	12.75	3.81	3.31
query30	3.02	2.95	2.97
query31	2.81	0.56	0.40
query32	3.24	0.58	0.51
query33	3.04	3.28	3.25
query34	15.98	5.42	5.06
query35	5.00	5.07	5.03
query36	0.69	0.52	0.51
query37	0.10	0.08	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 106.55 s
Total hot run time: 33.24 s

hello-stephen · 2025-07-29T15:28:26Z

FE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

vinlee19 · 2025-07-30T06:44:19Z

run nonConcurrent

zddr

LGTM

zddr · 2025-07-30T09:59:16Z

+                .collect(Collectors.toList());

        for (Partition partition : paimonPartitions) {
+            int index = 0;


can use
for(int i=0;i<paimonPartitions.size();i++)
instead of index

After careful consideration, a better approach would be to create a name-to-type mapping from partitionColumns, then look up the type directly by column name when iterating through partition.spec(). This eliminates the dependency on index-based matching and makes the code more robust.

github-actions · 2025-07-30T10:03:04Z

PR approved by anyone and no changes requested.

vinlee19 · 2025-07-30T13:07:02Z

run buildall

doris-robot · 2025-07-30T13:45:13Z

TPC-H: Total hot run time: 33951 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a50be83779afc2900603ab1998a712d49d8ef2ee, data reload: false

------ Round 1 ----------------------------------
q1	17744	5717	5622	5622
q2	1938	275	186	186
q3	10313	1320	691	691
q4	10213	969	521	521
q5	7528	2338	2270	2270
q6	177	166	132	132
q7	893	749	618	618
q8	9304	1299	1070	1070
q9	6690	5024	5116	5024
q10	6883	2332	1968	1968
q11	450	276	277	276
q12	344	355	224	224
q13	17767	3465	2936	2936
q14	228	249	218	218
q15	559	475	466	466
q16	421	437	367	367
q17	557	835	356	356
q18	7327	6990	6993	6990
q19	2099	963	550	550
q20	331	317	213	213
q21	2781	3028	2253	2253
q22	1071	1023	1000	1000
Total cold run time: 105618 ms
Total hot run time: 33951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5945	5839	5727	5727
q2	229	300	215	215
q3	2078	2568	2200	2200
q4	1273	1716	1328	1328
q5	4111	4472	4248	4248
q6	230	183	157	157
q7	2119	1958	1849	1849
q8	2684	2547	2452	2452
q9	7818	7387	7511	7387
q10	3211	3423	2940	2940
q11	559	568	516	516
q12	714	770	639	639
q13	3482	3778	3195	3195
q14	292	298	277	277
q15	509	447	448	447
q16	451	505	470	470
q17	1194	1372	1321	1321
q18	8161	7942	7733	7733
q19	12959	991	979	979
q20	2809	2050	1794	1794
q21	14821	4180	4316	4180
q22	1068	1052	997	997
Total cold run time: 76717 ms
Total hot run time: 51051 ms

doris-robot · 2025-07-30T13:57:03Z

TPC-DS: Total hot run time: 161969 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a50be83779afc2900603ab1998a712d49d8ef2ee, data reload: false

reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
============================================
query1	1010	375	413	375
query2	6529	1726	1692	1692
query3	6742	225	218	218
query4	26963	23728	22856	22856
query5	4407	608	502	502
query6	302	235	237	235
query7	4637	532	294	294
query8	282	253	228	228
query9	8571	3191	3148	3148
query10	476	327	289	289
query11	15827	14970	15107	14970
query12	181	133	130	130
query13	1665	548	427	427
query14	9437	7605	7833	7605
query15	223	199	162	162
query16	7135	640	464	464
query17	978	764	644	644
query18	2020	455	328	328
query19	222	219	203	203
query20	165	152	136	136
query21	217	131	107	107
query22	3869	4044	3930	3930
query23	34393	33966	34182	33966
query24	7562	2371	2377	2371
query25	533	498	436	436
query26	720	286	159	159
query27	2422	497	362	362
query28	3033	2260	2259	2259
query29	650	596	482	482
query30	307	227	203	203
query31	874	788	697	697
query32	87	77	80	77
query33	537	405	381	381
query34	819	844	505	505
query35	825	826	739	739
query36	1014	1027	930	930
query37	134	109	90	90
query38	3993	4047	3987	3987
query39	1443	1371	1347	1347
query40	239	140	130	130
query41	63	58	54	54
query42	136	123	137	123
query43	524	503	468	468
query44	1422	883	870	870
query45	209	188	182	182
query46	940	1063	667	667
query47	1792	1818	1774	1774
query48	402	417	324	324
query49	683	511	406	406
query50	691	677	415	415
query51	4198	4168	4291	4168
query52	133	127	123	123
query53	265	289	212	212
query54	647	630	558	558
query55	92	84	88	84
query56	370	357	348	348
query57	1183	1215	1128	1128
query58	343	340	331	331
query59	2583	2527	2464	2464
query60	433	396	387	387
query61	131	121	123	121
query62	776	746	678	678
query63	247	213	222	213
query64	2444	1068	837	837
query65	4138	4094	4082	4082
query66	1019	456	328	328
query67	query68	16734	874	850	850
query69	1147	269	287	269
query70	1321	1118	1126	1118
query71	791	313	315	313
query72	9187	2252	2148	2148
query73	3365	690	354	354
query74	9056	8967	8462	8462
query75	7636	3135	2651	2651
query76	8875	1208	781	781
query77	1139	429	336	336
query78	9550	11173	query79	15655	664	595	595
query80	3384	595	480	480
query81	600	271	252	252
query82	553	153	116	116
query83	403	289	269	269
query84	298	100	87	87
query85	943	369	330	330
query86	355	324	311	311
query87	4263	4299	4253	4253
query88	5579	2252	2242	2242
query89	490	371	323	323
query90	2495	227	225	225
query91	140	146	109	109
query92	85	70	66	66
query93	6676	1025	671	671
query94	1397	395	284	284
query95	432	323	329	323
query96	506	595	284	284
query97	2684	2720	2556	2556
query98	256	233	230	230
query99	1480	1338	1272	1272
Total cold run time: 295232 ms
Total hot run time: 161969 ms

doris-robot · 2025-07-30T14:02:29Z

ClickBench: Total hot run time: 32.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a50be83779afc2900603ab1998a712d49d8ef2ee, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.05	0.04
query3	0.24	0.08	0.07
query4	1.62	0.10	0.10
query5	0.40	0.41	0.42
query6	1.18	0.67	0.67
query7	0.02	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.49	0.46
query10	0.53	0.52	0.52
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.64	0.65	0.62
query14	0.88	1.23	0.95
query15	0.94	0.88	0.88
query16	0.39	0.39	0.39
query17	1.04	1.07	1.05
query18	0.21	0.22	0.20
query19	1.95	1.77	1.85
query20	0.02	0.01	0.01
query21	15.43	0.86	0.55
query22	0.76	1.22	0.62
query23	14.98	1.17	0.63
query24	6.50	0.64	0.57
query25	0.49	0.17	0.08
query26	0.61	0.15	0.13
query27	0.06	0.05	0.05
query28	9.80	0.87	0.44
query29	12.61	3.84	3.33
query30	3.11	2.97	2.91
query31	2.82	0.57	0.40
query32	3.24	0.57	0.50
query33	3.07	3.14	3.17
query34	15.92	5.37	4.89
query35	4.88	4.92	4.94
query36	0.72	0.52	0.50
query37	0.09	0.07	0.07
query38	0.06	0.06	0.05
query39	0.03	0.03	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.05	0.02	0.02
query43	0.05	0.03	0.04
Total cold run time: 106.65 s
Total hot run time: 32.45 s

hello-stephen · 2025-07-30T14:08:14Z

FE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-08-01T06:24:27Z

PR approved by at least one committer and no changes requested.

vinlee19 · 2025-08-01T14:43:35Z

run cloud_p0

### What problem does this PR solve? **Background**: PR [#46641](#46641) worked around Paimon date partition failures by treating date partitions as unpartitioned tables. **Root Cause**: Paimon stores date partition values as integers (e.g., `20231225` for `2023-12-25`), but the previous implementation failed to convert these to Doris-compatible date formats, resulting in: ``` org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = date literal [19737] is invalid: null ```

…#54043 (#54250) Cherry-picked from #54043 Co-authored-by: Petrichor <xiaowenli@selectdb.com>

…3789) ### What problem does this PR solve? Implement show partition feature. Prerequisites: PR #54043 date conversion fixes.

vinlee19 added 4 commits July 21, 2025 21:03

show partitions command for paimon

b078da1

Revert "show partitions command for paimon"

4d45f4e

This reverts commit b078da1.

add mvmt regression

bc47b1e

fix can not convert paimon date partition

8cd1ae6

vinlee19 mentioned this pull request Jul 29, 2025

[feat][paimon] support show partitions command for paimon catalog #53789

Merged

16 tasks

zddr approved these changes Jul 30, 2025

View reviewed changes

github-actions Bot added the reviewed label Jul 30, 2025

optimize get column name logic

a50be83

morningman added dev/2.1.x dev/3.0.x dev/3.1.x labels Aug 1, 2025

morningman approved these changes Aug 1, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2025

zddr approved these changes Aug 1, 2025

View reviewed changes

morningman merged commit 1b7f43d into apache:master Aug 3, 2025
33 checks passed

github-actions Bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Aug 3, 2025

github-actions Bot mentioned this pull request Aug 3, 2025

branch-3.1: [fix][paimon] Fix Paimon date partition conversion failure #54043 #54250

Merged

morrySnow pushed a commit that referenced this pull request Aug 4, 2025

branch-3.1: [fix][paimon] Fix Paimon date partition conversion failure …

f14b453

…#54043 (#54250) Cherry-picked from #54043 Co-authored-by: Petrichor <xiaowenli@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Aug 4, 2025

morningman removed dev/2.1.x dev/3.0.x dev/3.0.x-conflict dev/2.1.x-conflict labels Aug 11, 2025

morningman pushed a commit that referenced this pull request Aug 11, 2025

[feat][paimon] support show partitions command for paimon catalog (#5…

48e0ef1

…3789) ### What problem does this PR solve? Implement show partition feature. Prerequisites: PR #54043 date conversion fixes.

Conversation

vinlee19 commented Jul 29, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 29, 2025

Uh oh!

vinlee19 commented Jul 29, 2025

Uh oh!

vinlee19 commented Jul 29, 2025

Uh oh!

doris-robot commented Jul 29, 2025

Uh oh!

doris-robot commented Jul 29, 2025

Uh oh!

doris-robot commented Jul 29, 2025

Uh oh!

hello-stephen commented Jul 29, 2025

FE UT Coverage Report

Uh oh!

vinlee19 commented Jul 30, 2025

Uh oh!

zddr left a comment

Choose a reason for hiding this comment

Uh oh!

zddr Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

vinlee19 Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 30, 2025

Uh oh!

vinlee19 commented Jul 30, 2025

Uh oh!

doris-robot commented Jul 30, 2025

Uh oh!

doris-robot commented Jul 30, 2025

Uh oh!

doris-robot commented Jul 30, 2025

Uh oh!

hello-stephen commented Jul 30, 2025

FE UT Coverage Report

Uh oh!

github-actions Bot commented Aug 1, 2025

Uh oh!

vinlee19 commented Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants