Skip to content

branch-3.1: [fix](load) fix missing error url return for stream load #54115#54267

Merged
morrySnow merged 1 commit intobranch-3.1from
auto-pick-54115-branch-3.1
Aug 4, 2025
Merged

branch-3.1: [fix](load) fix missing error url return for stream load #54115#54267
morrySnow merged 1 commit intobranch-3.1from
auto-pick-54115-branch-3.1

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

@github-actions github-actions bot commented Aug 4, 2025

Cherry-picked from #54115

@liaoxin01
[fix](load) fix missing error url return for stream load (#54115)
3fc6993 
when pipe is cancelled, stream load execution may return early without
error url.
(test_stream_load_with_filtered_rows.groovy:64) - Stream load result: {
    "TxnId": 11,
    "Label": "2bbde37b-0589-4cce-8497-58e25af46590",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[CANCELLED]cancelled: [DATA_QUALITY_ERROR]Encountered
unqualified data, stop processing. Please check if the source data
matches the schema, and consider disabling strict mode or increasing
max_filter_ratio.. cur path: ",
    "NumberTotalRows": 32512,
    "NumberLoadedRows": 32460,
    "NumberFilteredRows": 52,
    "NumberUnselectedRows": 0,
    "LoadBytes": 29494781,
    "LoadTimeMs": 2138,
    "BeginTxnTimeMs": 1,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 24,
    "WriteDataTimeMs": 0,
    "ReceiveDataTimeMs": 1836,
    "CommitAndPublishTimeMs": 0
}
@github-actions github-actions bot requested a review from morrySnow as a code owner August 4, 2025 03:40
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Aug 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Aug 4, 2025
@dataroaring dataroaring reopened this Aug 4, 2025
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Aug 4, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.40% (12639/27838)
Line Coverage 36.27% (112697/310730)
Region Coverage 35.30% (58229/164943)
Branch Coverage 32.50% (31669/97448)

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 4, 2025

TPC-H: Total hot run time: 32995 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3fc699301afec7ca6c3c9b23fa336a40e33b123e, data reload: false

------ Round 1 ----------------------------------
q1	17729	5562	5441	5441
q2	2061	284	177	177
q3	10664	1269	768	768
q4	10220	888	472	472
q5	8309	2364	2202	2202
q6	183	165	131	131
q7	946	756	620	620
q8	9341	1451	1258	1258
q9	5256	4972	4915	4915
q10	6798	2281	1843	1843
q11	476	299	271	271
q12	357	365	213	213
q13	17793	3644	3000	3000
q14	246	229	209	209
q15	537	474	475	474
q16	432	420	367	367
q17	593	873	360	360
q18	6918	6443	6445	6443
q19	1566	963	587	587
q20	338	340	207	207
q21	2988	2238	1999	1999
q22	1060	1066	1038	1038
Total cold run time: 104811 ms
Total hot run time: 32995 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5657	5582	5572	5572
q2	253	329	244	244
q3	2231	2608	2328	2328
q4	1366	1872	1378	1378
q5	4401	4956	4975	4956
q6	168	165	130	130
q7	2064	1948	1859	1859
q8	2600	2840	2773	2773
q9	7263	7172	7218	7172
q10	3030	3347	2812	2812
q11	564	502	497	497
q12	599	719	570	570
q13	3406	3811	3238	3238
q14	289	290	281	281
q15	525	475	476	475
q16	449	479	444	444
q17	1260	1744	1250	1250
q18	7512	7450	7318	7318
q19	841	1192	1074	1074
q20	2004	2049	1938	1938
q21	5248	4965	4656	4656
q22	1115	1123	1019	1019
Total cold run time: 52845 ms
Total hot run time: 51984 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 4, 2025

TPC-DS: Total hot run time: 197541 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3fc699301afec7ca6c3c9b23fa336a40e33b123e, data reload: false

query1	1337	922	910	910
query2	6228	1948	1893	1893
query3	10899	4351	4509	4351
query4	32764	23888	23477	23477
query5	3716	609	466	466
query6	283	206	172	172
query7	3988	528	329	329
query8	296	244	235	235
query9	9533	2588	2580	2580
query10	483	348	275	275
query11	17985	15690	15452	15452
query12	170	112	107	107
query13	1556	549	461	461
query14	9887	6852	6953	6852
query15	248	204	190	190
query16	7934	693	519	519
query17	1614	766	610	610
query18	2172	418	340	340
query19	201	205	182	182
query20	125	120	120	120
query21	209	128	109	109
query22	4564	4506	4359	4359
query23	35058	34400	34485	34400
query24	7557	2738	2799	2738
query25	534	493	431	431
query26	748	291	190	190
query27	2271	484	360	360
query28	5007	2243	2194	2194
query29	729	616	477	477
query30	245	185	166	166
query31	978	921	860	860
query32	92	67	66	66
query33	543	384	321	321
query34	745	848	536	536
query35	777	804	730	730
query36	990	1076	962	962
query37	118	92	68	68
query38	4011	4100	4003	4003
query39	1534	1484	1456	1456
query40	209	120	107	107
query41	52	53	47	47
query42	128	109	104	104
query43	516	544	483	483
query44	1412	844	817	817
query45	184	180	179	179
query46	935	1057	695	695
query47	1936	1972	1934	1934
query48	407	429	338	338
query49	791	534	407	407
query50	704	742	446	446
query51	7385	7399	7132	7132
query52	104	103	92	92
query53	235	278	192	192
query54	551	552	477	477
query55	79	81	83	81
query56	283	281	274	274
query57	1269	1279	1212	1212
query58	242	224	242	224
query59	2987	3165	2909	2909
query60	301	277	285	277
query61	171	128	112	112
query62	793	733	727	727
query63	234	203	204	203
query64	3307	1019	689	689
query65	3387	3323	3321	3321
query66	757	416	310	310
query67	16279	15732	15427	15427
query68	7811	860	533	533
query69	492	300	264	264
query70	1170	1118	1152	1118
query71	404	312	269	269
query72	5906	3809	3806	3806
query73	649	765	350	350
query74	10570	9103	9307	9103
query75	3250	3179	2696	2696
query76	3082	1184	780	780
query77	473	370	295	295
query78	10392	10446	9624	9624
query79	3167	858	614	614
query80	765	532	437	437
query81	502	259	232	232
query82	649	121	88	88
query83	164	162	150	150
query84	248	105	83	83
query85	808	378	300	300
query86	396	321	285	285
query87	4312	4284	4243	4243
query88	5174	2395	2390	2390
query89	415	335	297	297
query90	1820	194	196	194
query91	150	136	111	111
query92	73	59	54	54
query93	2150	917	543	543
query94	704	434	292	292
query95	343	280	275	275
query96	493	613	283	283
query97	3181	3272	3178	3178
query98	236	215	199	199
query99	1627	1411	1292	1292
Total cold run time: 295343 ms
Total hot run time: 197541 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 4, 2025

ClickBench: Total hot run time: 29.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3fc699301afec7ca6c3c9b23fa336a40e33b123e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.53	0.51	0.51
query6	1.13	0.74	0.74
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.56	0.51	0.49
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.62	0.60	0.61
query14	0.78	0.78	0.80
query15	0.84	0.84	0.83
query16	0.37	0.38	0.41
query17	1.07	1.05	1.04
query18	0.25	0.21	0.21
query19	1.92	1.86	1.91
query20	0.02	0.02	0.01
query21	15.38	0.90	0.60
query22	0.74	0.78	0.61
query23	15.09	1.40	0.61
query24	3.03	1.48	1.33
query25	0.12	0.15	0.16
query26	0.39	0.15	0.13
query27	0.05	0.05	0.05
query28	14.21	1.05	0.44
query29	12.55	4.00	3.28
query30	0.25	0.09	0.06
query31	2.82	0.62	0.38
query32	3.22	0.54	0.46
query33	2.97	2.98	3.03
query34	16.82	5.17	4.56
query35	4.61	4.57	4.55
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.37 s
Total hot run time: 29.45 s

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.13% (20941/27506)
Line Coverage 69.55% (215967/310523)
Region Coverage 67.58% (129264/191281)
Branch Coverage 61.19% (67295/109972)

@morrySnow morrySnow merged commit 7ce9725 into branch-3.1 Aug 4, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-54115-branch-3.1 branch August 4, 2025 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @liaoxin01