[Fix](tvf) Fix that tvf reading empty files in compressed formats. by BePPPower · Pull Request #35141 · apache/doris

BePPPower · 2024-05-21T10:10:55Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-21T10:11:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

BePPPower · 2024-05-21T10:11:04Z

run buildall

doris-robot · 2024-05-21T10:36:59Z

TeamCity be ut coverage result:
Function Coverage: 37.80% (8077/21369)
Line Coverage: 29.45% (65951/223919)
Region Coverage: 28.92% (33956/117395)
Branch Coverage: 24.78% (17425/70320)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d_5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d/report/index.html

doris-robot · 2024-05-21T11:02:33Z

TPC-H: Total hot run time: 49985 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d, data reload: false

------ Round 1 ----------------------------------
q1	17892	6240	4368	4368
q2	2055	155	148	148
q3	10520	2005	1961	1961
q4	10376	1283	1349	1283
q5	8480	4016	3912	3912
q6	238	124	126	124
q7	2026	1613	1605	1605
q8	9538	2765	2744	2744
q9	13372	10313	10272	10272
q10	8614	3536	3516	3516
q11	416	249	255	249
q12	475	307	319	307
q13	18438	4007	4048	4007
q14	361	340	329	329
q15	518	464	451	451
q16	690	581	574	574
q17	1173	924	913	913
q18	7329	7022	6849	6849
q19	1756	1569	1573	1569
q20	531	309	285	285
q21	4500	4221	4121	4121
q22	501	398	409	398
Total cold run time: 119799 ms
Total hot run time: 49985 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4332	4288	4288
q2	324	225	224	224
q3	4190	4181	4147	4147
q4	2744	2745	2766	2745
q5	7145	7195	7094	7094
q6	239	121	122	121
q7	3308	2806	2785	2785
q8	4400	4503	4543	4503
q9	16897	16701	16777	16701
q10	4251	4283	4257	4257
q11	756	683	687	683
q12	1050	866	834	834
q13	6069	3741	3758	3741
q14	452	427	441	427
q15	516	470	453	453
q16	727	702	681	681
q17	3836	3869	3910	3869
q18	8781	8717	8832	8717
q19	1737	1689	1689	1689
q20	2379	2135	2131	2131
q21	8519	8573	8479	8479
q22	1037	939	929	929
Total cold run time: 83694 ms
Total hot run time: 79498 ms

doris-robot · 2024-05-21T11:15:19Z

TPC-DS: Total hot run time: 203439 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d, data reload: false

query1	953	396	381	381
query2	6548	2775	2887	2775
query3	6921	210	211	210
query4	21070	18015	18101	18015
query5	19717	6510	6557	6510
query6	285	214	223	214
query7	4156	299	310	299
query8	241	284	258	258
query9	3135	2715	2635	2635
query10	418	317	283	283
query11	11346	10671	10553	10553
query12	123	84	76	76
query13	5586	673	692	673
query14	17958	13278	13373	13278
query15	357	222	223	222
query16	6481	275	260	260
query17	1722	1444	857	857
query18	2349	409	400	400
query19	218	150	149	149
query20	75	78	79	78
query21	193	95	94	94
query22	5227	5056	4977	4977
query23	32426	32023	32122	32023
query24	7713	6533	6511	6511
query25	532	418	420	418
query26	713	163	162	162
query27	2176	300	301	300
query28	6178	2352	2321	2321
query29	2868	2771	2780	2771
query30	239	160	166	160
query31	931	739	721	721
query32	72	62	60	60
query33	406	267	250	250
query34	854	463	493	463
query35	1094	913	917	913
query36	1447	1258	1060	1060
query37	90	58	65	58
query38	3094	2949	2878	2878
query39	1386	1331	1347	1331
query40	304	103	92	92
query41	37	39	35	35
query42	81	85	84	84
query43	807	626	701	626
query44	1159	721	718	718
query45	247	227	234	227
query46	1222	946	963	946
query47	1754	1716	1713	1713
query48	1025	707	700	700
query49	640	383	379	379
query50	847	610	599	599
query51	4797	4645	4632	4632
query52	91	79	91	79
query53	449	317	318	317
query54	2677	2456	2452	2452
query55	87	88	86	86
query56	230	205	215	205
query57	1214	1097	1120	1097
query58	215	206	217	206
query59	4299	4238	4150	4150
query60	205	198	195	195
query61	86	84	83	83
query62	804	519	492	492
query63	492	346	339	339
query64	2766	1522	1458	1458
query65	3642	3597	3579	3579
query66	780	374	365	365
query67	17344	15348	15471	15348
query68	8699	644	671	644
query69	568	354	362	354
query70	1777	1305	1348	1305
query71	406	313	324	313
query72	6482	3569	3427	3427
query73	726	322	323	322
query74	6293	5864	5869	5864
query75	5253	3800	3765	3765
query76	5182	1167	1132	1132
query77	902	263	251	251
query78	12663	11643	11476	11476
query79	9689	663	670	663
query80	1566	389	383	383
query81	493	239	230	230
query82	1568	105	93	93
query83	178	130	141	130
query84	262	71	69	69
query85	894	302	306	302
query86	338	309	289	289
query87	3202	3055	3032	3032
query88	5075	2375	2362	2362
query89	468	289	295	289
query90	1893	217	202	202
query91	188	135	136	135
query92	58	50	53	50
query93	6420	569	575	569
query94	693	200	209	200
query95	1113	1060	1059	1059
query96	654	338	331	331
query97	6502	6398	6413	6398
query98	191	176	167	167
query99	2987	834	934	834
Total cold run time: 318982 ms
Total hot run time: 203439 ms

doris-robot · 2024-05-21T11:20:50Z

ClickBench: Total hot run time: 31.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.03	0.02
query3	0.25	0.05	0.05
query4	1.79	0.07	0.07
query5	0.57	0.54	0.53
query6	1.25	0.62	0.64
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.51	0.47	0.48
query10	0.55	0.54	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.10
query13	0.62	0.61	0.62
query14	0.78	0.79	0.81
query15	0.81	0.79	0.77
query16	0.40	0.39	0.38
query17	1.01	1.05	1.03
query18	0.23	0.26	0.25
query19	1.94	1.82	1.87
query20	0.01	0.01	0.01
query21	15.48	0.57	0.54
query22	2.03	2.73	2.20
query23	17.23	1.10	1.08
query24	4.46	1.00	3.18
query25	0.39	0.13	0.04
query26	0.63	0.16	0.14
query27	0.04	0.05	0.04
query28	6.50	0.80	0.76
query29	12.68	2.33	2.39
query30	0.59	0.53	0.54
query31	2.81	0.40	0.39
query32	3.34	0.52	0.51
query33	3.20	3.16	3.12
query34	15.25	4.89	4.84
query35	4.89	4.90	4.89
query36	1.07	1.02	1.04
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.07 s
Total hot run time: 31.87 s

doris-robot · 2024-05-21T11:24:02Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5d5798c140a1e6c2dc5a9cfcf836cc94b8b6545d with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

BePPPower · 2024-05-21T12:43:39Z

run feut

…pache#34926) 1. Fix the issue with tvf reading empty compressed files. 2. move two test cases (`test_local_tvf_compression` and `test_s3_tvf_compression`) from p2 to p0

morningman · 2024-05-22T08:00:06Z

run buildall

doris-robot · 2024-05-22T08:32:52Z

TeamCity be ut coverage result:
Function Coverage: 37.79% (8076/21369)
Line Coverage: 29.45% (65934/223919)
Region Coverage: 28.92% (33946/117395)
Branch Coverage: 24.77% (17417/70320)
Coverage Report: http://coverage.selectdb-in.cc/coverage/76450329cad61dd14297bedd532ef424de64659a_76450329cad61dd14297bedd532ef424de64659a/report/index.html

doris-robot · 2024-05-22T08:35:42Z

TPC-H: Total hot run time: 50004 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76450329cad61dd14297bedd532ef424de64659a, data reload: false

------ Round 1 ----------------------------------
q1	17657	4362	4341	4341
q2	2058	150	142	142
q3	10277	1919	1982	1919
q4	10350	1231	1327	1231
q5	9011	3951	3932	3932
q6	236	123	123	123
q7	2040	1578	1597	1578
q8	9282	2759	2690	2690
q9	10797	10547	10461	10461
q10	8950	3536	3504	3504
q11	425	232	249	232
q12	469	299	308	299
q13	18372	4051	4038	4038
q14	357	334	329	329
q15	520	479	468	468
q16	668	585	568	568
q17	1140	950	972	950
q18	7324	7028	6909	6909
q19	1710	1551	1532	1532
q20	537	318	311	311
q21	4525	4139	4058	4058
q22	493	389	395	389
Total cold run time: 117198 ms
Total hot run time: 50004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4324	4340	4324
q2	327	233	221	221
q3	4233	4145	4186	4145
q4	2768	2752	2789	2752
q5	7464	7416	7439	7416
q6	243	123	128	123
q7	3348	3044	2918	2918
q8	4519	4537	4521	4521
q9	17662	17115	17314	17115
q10	4215	4244	4269	4244
q11	753	665	693	665
q12	1034	846	842	842
q13	7120	3751	3751	3751
q14	452	433	431	431
q15	508	453	456	453
q16	756	680	671	671
q17	3824	3840	3819	3819
q18	8858	8672	8660	8660
q19	1704	1699	1633	1633
q20	2367	2099	2117	2099
q21	8580	8387	8614	8387
q22	1038	933	932	932
Total cold run time: 86114 ms
Total hot run time: 80122 ms

doris-robot · 2024-05-22T08:48:23Z

TPC-DS: Total hot run time: 202920 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76450329cad61dd14297bedd532ef424de64659a, data reload: false

query1	928	387	380	380
query2	6539	2699	2580	2580
query3	6918	209	209	209
query4	20910	17949	18015	17949
query5	19721	6558	6535	6535
query6	303	225	227	225
query7	4152	300	313	300
query8	256	249	279	249
query9	3137	2711	2641	2641
query10	421	280	313	280
query11	11254	10745	10738	10738
query12	128	78	74	74
query13	5581	675	680	675
query14	17757	13292	13336	13292
query15	356	227	230	227
query16	6468	272	269	269
query17	1723	1443	859	859
query18	2334	405	403	403
query19	206	158	152	152
query20	83	77	83	77
query21	187	97	89	89
query22	5293	5129	5043	5043
query23	32487	31963	31918	31918
query24	6949	6551	6420	6420
query25	517	428	423	423
query26	523	161	160	160
query27	1829	293	296	293
query28	6162	2373	2324	2324
query29	2924	2864	2744	2744
query30	245	165	164	164
query31	903	741	721	721
query32	63	64	59	59
query33	400	249	256	249
query34	844	469	490	469
query35	1139	926	922	922
query36	1430	1164	1092	1092
query37	90	59	58	58
query38	3076	2952	2937	2937
query39	1371	1328	1321	1321
query40	208	93	96	93
query41	39	40	33	33
query42	89	81	84	81
query43	725	697	653	653
query44	1148	725	721	721
query45	245	229	231	229
query46	1216	954	986	954
query47	1819	1739	1833	1739
query48	1000	720	715	715
query49	616	369	372	369
query50	858	591	598	591
query51	4767	4701	4676	4676
query52	103	77	99	77
query53	443	324	318	318
query54	2625	2473	2508	2473
query55	83	76	75	75
query56	219	208	215	208
query57	1145	1088	1137	1088
query58	223	200	193	193
query59	3966	3793	3674	3674
query60	202	189	202	189
query61	92	84	85	84
query62	788	474	468	468
query63	482	342	341	341
query64	2526	1546	1540	1540
query65	3654	3542	3547	3542
query66	759	387	378	378
query67	15402	15045	14905	14905
query68	10227	654	651	651
query69	570	353	364	353
query70	1988	1304	1578	1304
query71	419	317	317	317
query72	6475	3476	3440	3440
query73	731	337	322	322
query74	6321	5849	5815	5815
query75	5367	3779	3716	3716
query76	6546	1186	1195	1186
query77	1122	256	253	253
query78	13076	11966	12257	11966
query79	8124	629	658	629
query80	785	397	390	390
query81	493	232	240	232
query82	1395	95	97	95
query83	171	134	132	132
query84	262	67	69	67
query85	865	329	307	307
query86	326	288	338	288
query87	3224	3004	3024	3004
query88	4561	2353	2361	2353
query89	452	276	271	271
query90	1967	205	211	205
query91	167	136	147	136
query92	60	51	51	51
query93	6184	550	565	550
query94	658	202	206	202
query95	1119	1056	1042	1042
query96	659	333	330	330
query97	6607	6411	6359	6359
query98	189	167	182	167
query99	2993	860	876	860
Total cold run time: 315152 ms
Total hot run time: 202920 ms

doris-robot · 2024-05-22T08:53:52Z

ClickBench: Total hot run time: 30.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76450329cad61dd14297bedd532ef424de64659a, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.05	0.05
query4	1.80	0.06	0.05
query5	0.53	0.53	0.52
query6	1.24	0.66	0.63
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.49	0.48
query10	0.55	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.78	0.78	0.79
query15	0.79	0.75	0.76
query16	0.37	0.37	0.37
query17	0.99	1.01	1.02
query18	0.21	0.28	0.22
query19	1.91	1.87	1.87
query20	0.02	0.01	0.01
query21	15.58	0.56	0.55
query22	2.27	2.28	1.78
query23	17.33	1.05	0.94
query24	8.02	1.45	0.83
query25	0.38	0.13	0.06
query26	0.87	0.18	0.15
query27	0.04	0.04	0.03
query28	4.86	0.76	0.77
query29	12.60	2.37	2.34
query30	0.56	0.54	0.53
query31	2.80	0.39	0.37
query32	3.37	0.50	0.50
query33	3.09	3.08	3.07
query34	15.25	4.83	4.84
query35	4.89	4.88	4.84
query36	1.04	1.02	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.29 s
Total hot run time: 30.97 s

doris-robot · 2024-05-22T08:57:07Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 76450329cad61dd14297bedd532ef424de64659a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

morningman

LGTM

github-actions · 2024-05-22T10:56:27Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-22T10:56:29Z

PR approved by anyone and no changes requested.

…pache#34926) (apache#35141) bp: apache#34926

github-actions bot added the kind/test label May 21, 2024

[Fix](tvf) Fix that tvf reading empty files in compressed formats. (a…

7645032

…pache#34926) 1. Fix the issue with tvf reading empty compressed files. 2. move two test cases (`test_local_tvf_compression` and `test_s3_tvf_compression`) from p2 to p0

morningman force-pushed the fixEmptySnappy branch from 5d5798c to 7645032 Compare May 22, 2024 07:51

morningman approved these changes May 22, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 22, 2024

github-actions bot added the reviewed label May 22, 2024

morningman merged commit ca57582 into apache:branch-2.0 May 22, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[Fix](tvf) Fix that tvf reading empty files in compressed formats. (a…

0289637

…pache#34926) (apache#35141) bp: apache#34926

Conversation

BePPPower commented May 21, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

BePPPower commented May 21, 2024

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

doris-robot commented May 21, 2024

Uh oh!

BePPPower commented May 21, 2024

Uh oh!

morningman commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 22, 2024

Uh oh!

github-actions bot commented May 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants