Skip to content

[fix](hive)fix hive insert only translaction table.(#45753)(#46385)#46454

Merged
morningman merged 3 commits intoapache:branch-3.0from
hubgeter:pick_30_hive_trans_insert_only
Jan 7, 2025
Merged

[fix](hive)fix hive insert only translaction table.(#45753)(#46385)#46454
morningman merged 3 commits intoapache:branch-3.0from
hubgeter:pick_30_hive_trans_insert_only

Conversation

@hubgeter
Copy link
Copy Markdown
Contributor

@hubgeter hubgeter commented Jan 6, 2025 •
edited
Loading

What problem does this PR solve?

bp #45753 : fix read hive insert only Transaction table.
bp #46385 , #45999 : fix #45753 case unstable.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Jan 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Copy Markdown
Contributor Author

hubgeter commented Jan 6, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

TPC-H: Total hot run time: 40950 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 865d8fa67e36ff3ab991ec9fe6c2b682bda912ba, data reload: false

------ Round 1 ----------------------------------
q1	17585	7439	7263	7263
q2	2039	166	150	150
q3	10575	1148	1184	1148
q4	10228	778	711	711
q5	7742	2920	2787	2787
q6	237	147	142	142
q7	960	603	599	599
q8	9354	1963	2044	1963
q9	6724	6422	6420	6420
q10	6993	2313	2295	2295
q11	464	274	272	272
q12	419	210	212	210
q13	17784	3018	2971	2971
q14	234	201	207	201
q15	578	534	511	511
q16	682	639	620	620
q17	974	548	549	548
q18	7266	6769	6689	6689
q19	1388	1043	1058	1043
q20	483	201	203	201
q21	3984	3288	3199	3199
q22	1087	1012	1007	1007
Total cold run time: 107780 ms
Total hot run time: 40950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7380	7220	7295	7220
q2	338	229	231	229
q3	2891	2971	2992	2971
q4	2025	1857	1783	1783
q5	5692	5748	5707	5707
q6	219	137	133	133
q7	2239	1829	1855	1829
q8	3421	3597	3561	3561
q9	8938	8995	8965	8965
q10	3612	3522	3528	3522
q11	585	510	525	510
q12	815	582	602	582
q13	15350	3117	3145	3117
q14	292	284	271	271
q15	570	524	523	523
q16	707	676	659	659
q17	1885	1610	1635	1610
q18	8252	7705	7693	7693
q19	1648	1387	1479	1387
q20	2055	1817	1799	1799
q21	5285	5193	5158	5158
q22	1110	1018	1002	1002
Total cold run time: 75309 ms
Total hot run time: 60231 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

TPC-DS: Total hot run time: 191289 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 865d8fa67e36ff3ab991ec9fe6c2b682bda912ba, data reload: false

query1	978	372	362	362
query2	6551	2118	2031	2031
query3	6709	214	223	214
query4	33955	23335	23493	23335
query5	4342	444	454	444
query6	292	193	180	180
query7	4630	317	315	315
query8	300	232	235	232
query9	9631	2682	2682	2682
query10	484	263	257	257
query11	18257	15340	15139	15139
query12	152	104	99	99
query13	1624	422	403	403
query14	10201	6680	7371	6680
query15	212	172	178	172
query16	7963	460	476	460
query17	1600	579	567	567
query18	2057	313	327	313
query19	233	160	160	160
query20	119	108	106	106
query21	213	106	102	102
query22	4603	4122	4164	4122
query23	34801	34250	33957	33957
query24	11333	2864	2888	2864
query25	692	416	415	415
query26	1406	172	174	172
query27	2935	346	350	346
query28	8138	2471	2433	2433
query29	940	478	444	444
query30	330	171	169	169
query31	1036	790	807	790
query32	97	59	60	59
query33	795	295	291	291
query34	953	503	525	503
query35	931	725	739	725
query36	1126	972	955	955
query37	148	87	74	74
query38	3972	3864	3863	3863
query39	1530	1434	1453	1434
query40	285	105	107	105
query41	53	51	52	51
query42	116	102	101	101
query43	539	500	500	500
query44	1261	803	802	802
query45	184	172	175	172
query46	1139	713	729	713
query47	1915	1828	1835	1828
query48	465	386	411	386
query49	1205	406	418	406
query50	814	409	414	409
query51	7057	7268	7183	7183
query52	105	94	92	92
query53	263	188	188	188
query54	1246	491	463	463
query55	77	83	83	83
query56	298	286	250	250
query57	1226	1151	1110	1110
query58	249	212	220	212
query59	3169	2985	3014	2985
query60	300	265	262	262
query61	135	134	139	134
query62	863	697	679	679
query63	230	200	199	199
query64	5562	656	628	628
query65	3330	3242	3198	3198
query66	1449	329	307	307
query67	15879	15610	15517	15517
query68	4617	577	571	571
query69	449	267	259	259
query70	1206	1129	1071	1071
query71	394	256	249	249
query72	6444	4012	4083	4012
query73	771	356	346	346
query74	10325	9020	8988	8988
query75	3424	2560	2679	2560
query76	2787	1042	1184	1042
query77	431	291	292	291
query78	10628	9738	9530	9530
query79	1594	574	603	574
query80	1080	438	423	423
query81	539	234	237	234
query82	953	118	117	117
query83	225	143	152	143
query84	243	84	80	80
query85	1348	296	290	290
query86	394	308	305	305
query87	4463	4333	4309	4309
query88	3676	2400	2350	2350
query89	401	297	290	290
query90	1982	183	188	183
query91	202	147	146	146
query92	61	52	50	50
query93	1219	551	549	549
query94	922	304	282	282
query95	354	253	255	253
query96	598	283	281	281
query97	3313	3260	3183	3183
query98	221	215	200	200
query99	1524	1296	1307	1296
Total cold run time: 302471 ms
Total hot run time: 191289 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

ClickBench: Total hot run time: 33.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 865d8fa67e36ff3ab991ec9fe6c2b682bda912ba, data reload: false

query1	0.03	0.04	0.02
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.50	0.52
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.03
query9	0.56	0.50	0.50
query10	0.55	0.55	0.54
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	2.94	3.03	2.90
query15	0.89	0.83	0.82
query16	0.38	0.38	0.37
query17	1.04	1.04	1.06
query18	0.23	0.22	0.22
query19	1.88	1.89	1.98
query20	0.02	0.01	0.01
query21	15.38	0.58	0.58
query22	2.82	2.17	1.43
query23	16.89	0.89	0.83
query24	2.70	1.40	1.61
query25	0.25	0.24	0.15
query26	0.53	0.14	0.13
query27	0.04	0.04	0.04
query28	10.15	1.11	1.05
query29	12.60	3.24	3.20
query30	0.25	0.06	0.06
query31	2.87	0.39	0.40
query32	3.22	0.47	0.47
query33	3.00	3.06	3.07
query34	17.04	4.52	4.48
query35	4.62	4.51	4.58
query36	0.67	0.48	0.50
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.6 s
Total hot run time: 33.12 s

@hubgeter
Copy link
Copy Markdown
Contributor Author

hubgeter commented Jan 6, 2025

run external

@hubgeter
[fix](hive)fix hive insert only translaction table. (apache#45753)
130006c 
bp apache#44001 , but no hive4 acid table.

Problem Summary:
1. Fixed the issue that when reading insert translaction only tables,
there was no acid check, which caused multiple data reads (i.e., reading
data from the previous base_n).
2. Forbidden to create, insert data, and delete aicd tables.
@hubgeter
[fix](regression)fix hive translation unstable case. (apache#46385)
dd869a1 
### What problem does this PR solve?

Problem Summary:
just fix case.
@hubgeter
[fix](case)fix acid regression case. (apache#45999)
86a94bb 
### What problem does this PR solve?

Problem Summary:
just fix error case.
@hubgeter hubgeter force-pushed the pick_30_hive_trans_insert_only branch from 3c3adda to 86a94bb Compare January 6, 2025 13:34
@hubgeter
Copy link
Copy Markdown
Contributor Author

hubgeter commented Jan 6, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

TPC-H: Total hot run time: 41263 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86a94bb7d58c2395de42c7f368a4c5d3e2c88535, data reload: false

------ Round 1 ----------------------------------
q1	17626	7439	7315	7315
q2	2049	178	182	178
q3	10521	1126	1232	1126
q4	10231	749	730	730
q5	7717	2886	2816	2816
q6	235	151	147	147
q7	987	611	613	611
q8	9355	1955	2021	1955
q9	6723	6448	6445	6445
q10	7018	2338	2343	2338
q11	467	272	265	265
q12	453	216	211	211
q13	17776	2984	3010	2984
q14	253	217	206	206
q15	578	524	521	521
q16	704	604	624	604
q17	989	593	586	586
q18	7380	6710	6737	6710
q19	1396	1124	1126	1124
q20	496	212	197	197
q21	4025	3394	3203	3203
q22	1094	991	993	991
Total cold run time: 108073 ms
Total hot run time: 41263 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7243	7230	7265	7230
q2	335	232	233	232
q3	2889	2790	2775	2775
q4	2017	1722	1682	1682
q5	5471	5480	5467	5467
q6	221	141	137	137
q7	2130	1735	1739	1735
q8	3251	3426	3441	3426
q9	8624	8610	8603	8603
q10	3502	3471	3452	3452
q11	587	496	516	496
q12	786	555	560	555
q13	10184	3019	3020	3019
q14	293	276	280	276
q15	563	503	516	503
q16	690	683	655	655
q17	1817	1563	1555	1555
q18	7591	7499	7215	7215
q19	1649	1538	1484	1484
q20	2028	1789	1802	1789
q21	5374	5126	5125	5125
q22	1101	997	986	986
Total cold run time: 68346 ms
Total hot run time: 58397 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

TPC-DS: Total hot run time: 193898 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86a94bb7d58c2395de42c7f368a4c5d3e2c88535, data reload: false

query1	971	417	382	382
query2	6427	2158	2126	2126
query3	6698	220	219	219
query4	33623	23711	23716	23711
query5	4372	462	445	445
query6	268	189	177	177
query7	4645	310	306	306
query8	287	228	234	228
query9	9454	2682	2660	2660
query10	487	248	265	248
query11	18354	15471	15456	15456
query12	158	105	105	105
query13	1659	444	415	415
query14	9514	7464	7399	7399
query15	233	172	184	172
query16	6616	440	431	431
query17	1581	589	573	573
query18	2028	336	336	336
query19	391	171	164	164
query20	124	112	110	110
query21	215	112	110	110
query22	4496	4410	4094	4094
query23	35381	34449	34202	34202
query24	11828	2910	2858	2858
query25	579	390	400	390
query26	831	174	168	168
query27	2872	351	356	351
query28	7667	2433	2414	2414
query29	714	451	424	424
query30	343	184	173	173
query31	1047	813	826	813
query32	72	59	59	59
query33	700	299	288	288
query34	916	503	532	503
query35	859	757	725	725
query36	1129	966	967	966
query37	218	75	74	74
query38	4085	3876	3933	3876
query39	1541	1456	1471	1456
query40	237	101	102	101
query41	52	49	48	48
query42	117	99	101	99
query43	548	506	488	488
query44	1276	795	809	795
query45	185	169	172	169
query46	1161	702	700	700
query47	1976	1846	1816	1816
query48	471	401	386	386
query49	1137	385	388	385
query50	829	414	412	412
query51	7348	7217	7189	7189
query52	103	90	88	88
query53	258	185	182	182
query54	1122	478	466	466
query55	80	79	79	79
query56	282	249	270	249
query57	1273	1125	1083	1083
query58	215	211	219	211
query59	3374	3131	2887	2887
query60	285	256	253	253
query61	113	109	115	109
query62	887	677	681	677
query63	227	193	200	193
query64	4106	657	651	651
query65	3465	3303	3306	3303
query66	1078	331	317	317
query67	16142	15689	15707	15689
query68	4895	566	562	562
query69	453	277	275	275
query70	1216	1144	1133	1133
query71	395	255	263	255
query72	5930	4166	4289	4166
query73	782	363	358	358
query74	10608	9074	8893	8893
query75	3515	2674	2644	2644
query76	2581	1010	1021	1010
query77	451	278	286	278
query78	11183	10022	9882	9882
query79	1086	604	603	603
query80	657	466	457	457
query81	537	257	254	254
query82	244	123	126	123
query83	207	147	151	147
query84	243	89	79	79
query85	987	331	303	303
query86	324	315	300	300
query87	4623	4401	4395	4395
query88	3749	2394	2368	2368
query89	408	290	293	290
query90	2002	186	190	186
query91	199	163	163	163
query92	62	54	53	53
query93	1060	557	544	544
query94	789	301	287	287
query95	356	251	254	251
query96	616	273	282	273
query97	3425	3236	3265	3236
query98	230	203	196	196
query99	1436	1324	1302	1302
Total cold run time: 296979 ms
Total hot run time: 193898 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jan 6, 2025

ClickBench: Total hot run time: 33.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86a94bb7d58c2395de42c7f368a4c5d3e2c88535, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.53	0.50	0.52
query6	1.12	0.74	0.73
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.57	0.49	0.50
query10	0.57	0.55	0.57
query11	0.14	0.10	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.60
query14	2.93	2.96	2.92
query15	0.88	0.83	0.83
query16	0.39	0.38	0.38
query17	1.07	1.06	1.04
query18	0.23	0.22	0.21
query19	2.00	1.95	1.92
query20	0.02	0.01	0.00
query21	15.36	0.60	0.56
query22	2.47	3.01	2.78
query23	17.05	0.90	0.83
query24	3.42	1.71	0.38
query25	0.35	0.09	0.07
query26	0.41	0.13	0.13
query27	0.04	0.05	0.04
query28	10.47	1.09	1.06
query29	12.58	3.30	3.27
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.25	0.46	0.46
query33	2.96	3.03	3.03
query34	17.19	4.50	4.44
query35	4.53	4.46	4.51
query36	0.68	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.59 s
Total hot run time: 33.37 s

@morningman morningman merged commit cda7023 into apache:branch-3.0 Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hubgeter @hello-stephen @doris-robot @morningman