[fix](stream load) do not throw exception but skip record when can not find database by sollhui · Pull Request #39360 · apache/doris

sollhui · 2024-08-14T11:08:31Z

When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory.

For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find database to solve this problem.

doris-robot · 2024-08-14T11:08:36Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-08-14T11:08:51Z

run buildall

sollhui · 2024-08-14T13:00:31Z

run buildall

sollhui · 2024-08-14T14:38:36Z

run buildall

doris-robot · 2024-08-14T14:56:30Z

TPC-H: Total hot run time: 37038 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

------ Round 1 ----------------------------------
q1	17605	4321	4279	4279
q2	2014	177	175	175
q3	10482	1194	1050	1050
q4	10141	740	703	703
q5	7744	2750	2691	2691
q6	219	135	140	135
q7	945	594	576	576
q8	9325	2037	2034	2034
q9	7242	6479	6498	6479
q10	6990	2210	2190	2190
q11	458	250	249	249
q12	386	223	218	218
q13	17761	3008	2968	2968
q14	279	244	227	227
q15	511	486	485	485
q16	491	399	382	382
q17	963	725	668	668
q18	7530	7013	6723	6723
q19	4381	1123	986	986
q20	704	324	323	323
q21	4212	2802	2509	2509
q22	1110	1014	988	988
Total cold run time: 111493 ms
Total hot run time: 37038 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4451	4244	4235	4235
q2	379	262	273	262
q3	2884	2606	2636	2606
q4	1906	1616	1661	1616
q5	5368	5349	5376	5349
q6	220	130	136	130
q7	2055	1633	1648	1633
q8	3158	3324	3313	3313
q9	8457	8382	8367	8367
q10	3393	3123	3134	3123
q11	591	490	502	490
q12	791	603	582	582
q13	17371	2982	2976	2976
q14	301	276	278	276
q15	523	481	476	476
q16	484	406	418	406
q17	1779	1495	1480	1480
q18	7774	7479	7389	7389
q19	1673	1465	1600	1465
q20	2008	1815	1794	1794
q21	5182	5104	5043	5043
q22	1078	987	1012	987
Total cold run time: 71826 ms
Total hot run time: 53998 ms

doris-robot · 2024-08-14T15:08:33Z

TPC-DS: Total hot run time: 184093 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

query1	908	374	362	362
query2	6451	1923	1847	1847
query3	6645	206	214	206
query4	28727	23244	22983	22983
query5	4215	518	491	491
query6	284	167	172	167
query7	4584	293	284	284
query8	237	194	195	194
query9	8470	2453	2398	2398
query10	426	260	261	260
query11	17633	14909	14999	14909
query12	151	97	108	97
query13	1629	382	365	365
query14	9795	7025	6530	6530
query15	219	170	173	170
query16	7803	493	505	493
query17	1584	581	555	555
query18	2026	298	288	288
query19	196	150	146	146
query20	115	105	109	105
query21	209	107	100	100
query22	4204	4170	3926	3926
query23	33689	33054	33115	33054
query24	12182	2892	2799	2799
query25	687	410	401	401
query26	1764	159	162	159
query27	2948	276	274	274
query28	7829	2039	2044	2039
query29	1068	429	424	424
query30	309	154	157	154
query31	977	735	783	735
query32	98	57	66	57
query33	792	292	301	292
query34	957	476	472	472
query35	815	743	715	715
query36	1050	925	925	925
query37	293	88	83	83
query38	3922	3800	3811	3800
query39	1430	1375	1386	1375
query40	274	122	121	121
query41	52	47	49	47
query42	119	100	105	100
query43	492	475	475	475
query44	1200	735	744	735
query45	191	164	162	162
query46	1109	780	717	717
query47	1874	1772	1798	1772
query48	360	291	292	291
query49	1208	429	430	429
query50	809	408	414	408
query51	6832	6687	6705	6687
query52	101	90	94	90
query53	266	184	188	184
query54	1007	471	461	461
query55	79	77	77	77
query56	292	269	265	265
query57	1148	1056	1081	1056
query58	239	239	243	239
query59	2958	2785	2663	2663
query60	300	277	279	277
query61	124	121	119	119
query62	838	659	642	642
query63	220	187	188	187
query64	6461	2241	1775	1775
query65	3190	3128	3152	3128
query66	1379	346	322	322
query67	15547	14775	14720	14720
query68	8772	573	585	573
query69	707	394	313	313
query70	1183	1021	1124	1021
query71	564	279	281	279
query72	7578	2243	2030	2030
query73	1854	319	325	319
query74	8959	8814	8776	8776
query75	5168	2664	2678	2664
query76	5071	1008	959	959
query77	776	334	312	312
query78	9791	9263	8982	8982
query79	8311	529	534	529
query80	1049	503	513	503
query81	593	228	225	225
query82	585	138	135	135
query83	316	147	144	144
query84	274	82	75	75
query85	1303	277	266	266
query86	351	289	261	261
query87	4356	4201	4232	4201
query88	3839	2294	2295	2294
query89	520	286	286	286
query90	2032	195	194	194
query91	123	96	96	96
query92	59	49	48	48
query93	6930	539	541	539
query94	854	346	293	293
query95	358	264	258	258
query96	629	268	266	266
query97	3213	2993	3009	2993
query98	209	204	212	204
query99	1541	1299	1272	1272
Total cold run time: 313858 ms
Total hot run time: 184093 ms

doris-robot · 2024-08-14T15:14:04Z

ClickBench: Total hot run time: 31.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.08	0.09
query5	0.50	0.50	0.49
query6	1.13	0.72	0.73
query7	0.01	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.49	0.46
query10	0.53	0.53	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.60	0.61	0.58
query14	0.76	0.79	0.77
query15	0.84	0.81	0.80
query16	0.36	0.37	0.37
query17	1.00	1.03	1.06
query18	0.22	0.23	0.21
query19	1.85	1.77	1.75
query20	0.01	0.01	0.02
query21	15.46	0.76	0.65
query22	4.03	7.14	2.42
query23	18.30	1.34	1.24
query24	2.08	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.24	1.02	1.01
query29	12.61	3.26	3.28
query30	0.23	0.06	0.05
query31	2.88	0.39	0.40
query32	3.27	0.48	0.49
query33	3.00	2.97	2.92
query34	17.05	4.30	4.35
query35	4.40	4.42	4.48
query36	0.65	0.49	0.47
query37	0.18	0.17	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.13	0.14
query41	0.09	0.05	0.06
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 31.15 s

liaoxin01

LGTM

github-actions · 2024-08-15T01:10:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-15T01:10:23Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

Jibing-Li · 2024-08-15T03:03:22Z

run buildall

doris-robot · 2024-08-15T03:23:08Z

TPC-H: Total hot run time: 37464 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

------ Round 1 ----------------------------------
q1	17663	4396	4322	4322
q2	2023	178	171	171
q3	10509	1121	1038	1038
q4	10147	708	686	686
q5	7758	2804	2765	2765
q6	225	140	144	140
q7	966	590	585	585
q8	9313	2021	2027	2021
q9	7232	6559	6512	6512
q10	7097	2210	2125	2125
q11	439	239	242	239
q12	393	224	220	220
q13	18834	2981	2985	2981
q14	271	242	234	234
q15	509	477	496	477
q16	496	393	388	388
q17	973	632	697	632
q18	7427	6775	6846	6775
q19	6427	1021	1062	1021
q20	721	349	328	328
q21	4637	2869	2809	2809
q22	1109	1010	995	995
Total cold run time: 115169 ms
Total hot run time: 37464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4528	4311	4325	4311
q2	373	283	276	276
q3	2872	2602	2585	2585
q4	1905	1603	1601	1601
q5	5460	5613	5666	5613
q6	230	138	138	138
q7	2113	1727	1720	1720
q8	3239	3437	3393	3393
q9	8821	8892	8865	8865
q10	3536	3311	3310	3310
q11	615	494	508	494
q12	762	617	598	598
q13	16678	3110	3168	3110
q14	341	275	292	275
q15	524	493	476	476
q16	497	428	427	427
q17	1847	1548	1520	1520
q18	8096	7821	7588	7588
q19	1685	1509	1527	1509
q20	2075	1792	1810	1792
q21	8416	5077	4929	4929
q22	1097	1020	1019	1019
Total cold run time: 75710 ms
Total hot run time: 55549 ms

doris-robot · 2024-08-15T03:34:56Z

TPC-DS: Total hot run time: 184553 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

query1	921	377	358	358
query2	6450	1895	1810	1810
query3	6665	212	220	212
query4	34149	23264	23091	23091
query5	4187	488	493	488
query6	261	162	171	162
query7	4584	291	288	288
query8	236	202	193	193
query9	8563	2455	2428	2428
query10	451	277	269	269
query11	17704	15016	15010	15010
query12	142	97	96	96
query13	1620	371	360	360
query14	9646	6982	7050	6982
query15	218	173	170	170
query16	7756	526	435	435
query17	1234	566	555	555
query18	1986	288	303	288
query19	205	146	146	146
query20	115	105	104	104
query21	207	108	99	99
query22	4424	4079	3924	3924
query23	33704	33242	33370	33242
query24	11618	2907	2854	2854
query25	627	396	390	390
query26	1066	158	158	158
query27	2248	281	280	280
query28	6913	2058	2053	2053
query29	882	419	430	419
query30	303	148	151	148
query31	970	741	765	741
query32	95	54	62	54
query33	773	286	287	286
query34	913	458	463	458
query35	850	725	746	725
query36	1058	895	937	895
query37	142	82	85	82
query38	3934	3907	3851	3851
query39	1425	1399	1393	1393
query40	211	111	116	111
query41	46	45	45	45
query42	121	94	102	94
query43	529	456	460	456
query44	1198	723	719	719
query45	196	163	163	163
query46	1099	779	714	714
query47	1862	1738	1779	1738
query48	362	294	287	287
query49	1010	423	415	415
query50	815	410	408	408
query51	6805	6622	6731	6622
query52	106	93	89	89
query53	260	195	186	186
query54	924	452	435	435
query55	77	75	74	74
query56	269	237	263	237
query57	1149	1060	1066	1060
query58	232	218	218	218
query59	2907	2799	2635	2635
query60	314	264	269	264
query61	101	94	97	94
query62	831	638	640	638
query63	216	184	186	184
query64	5065	2277	1756	1756
query65	3163	3109	3151	3109
query66	747	339	323	323
query67	15182	14677	14940	14677
query68	4515	536	540	536
query69	495	284	267	267
query70	1159	1147	1138	1138
query71	472	278	277	277
query72	7286	2269	1995	1995
query73	760	326	323	323
query74	9220	8765	8783	8765
query75	3933	2656	2679	2656
query76	3070	957	980	957
query77	727	310	315	310
query78	9842	9186	8914	8914
query79	2964	537	531	531
query80	1545	502	495	495
query81	596	223	224	223
query82	889	140	129	129
query83	357	149	149	149
query84	272	78	75	75
query85	2188	300	272	272
query86	501	292	302	292
query87	4469	4234	4259	4234
query88	4273	2329	2307	2307
query89	419	293	282	282
query90	1933	199	204	199
query91	130	103	97	97
query92	73	52	51	51
query93	4909	534	526	526
query94	909	292	302	292
query95	354	271	258	258
query96	609	321	269	269
query97	3212	3068	3010	3010
query98	220	204	197	197
query99	1512	1289	1279	1279
Total cold run time: 298855 ms
Total hot run time: 184553 ms

doris-robot · 2024-08-15T03:40:28Z

ClickBench: Total hot run time: 31.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95545681f927abaa44c383f9ab405e4019dea991, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.08
query5	0.50	0.50	0.48
query6	1.14	0.72	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.47	0.49
query10	0.54	0.53	0.54
query11	0.16	0.11	0.11
query12	0.16	0.12	0.13
query13	0.61	0.61	0.58
query14	0.76	0.78	0.78
query15	0.85	0.81	0.82
query16	0.35	0.35	0.38
query17	1.04	1.05	1.04
query18	0.22	0.22	0.22
query19	1.94	1.83	1.84
query20	0.01	0.00	0.01
query21	15.39	0.75	0.66
query22	3.83	6.72	2.26
query23	18.27	1.37	1.26
query24	2.12	0.22	0.20
query25	0.14	0.09	0.08
query26	0.30	0.22	0.21
query27	0.45	0.23	0.23
query28	13.26	1.03	1.02
query29	12.60	3.30	3.30
query30	0.23	0.06	0.04
query31	2.89	0.40	0.40
query32	3.26	0.49	0.50
query33	3.01	2.98	2.94
query34	16.98	4.33	4.32
query35	4.48	4.44	4.42
query36	0.66	0.49	0.48
query37	0.19	0.16	0.17
query38	0.16	0.15	0.15
query39	0.06	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.56 s
Total hot run time: 31.19 s

…t find database (#39360) When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory. For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory. This pr do not throw exception but skip record when can not find database to solve this problem.

…t find database (#39360) (#39531) pick (#39360) When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory. For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory. This pr do not throw exception but skip record when can not find database to solve this problem.

…t find database (#39360) (#39527) pick (#39360) When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory. For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory. This pr do not throw exception but skip record when can not find database to solve this problem.

…t find database (apache#39360) (apache#39531) pick (apache#39360) When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory. For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory. This pr do not throw exception but skip record when can not find database to solve this problem.

github-actions bot added the doing label Aug 14, 2024

do not throw exception but skip record when can not find database

9554568

liaoxin01 approved these changes Aug 15, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2024

github-actions bot added the reviewed label Aug 15, 2024

dataroaring approved these changes Aug 15, 2024

View reviewed changes

dataroaring added dev/2.1.x dev/2.0.x dev/3.0.x labels Aug 15, 2024

liaoxin01 merged commit e2c9280 into apache:master Aug 15, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

This was referenced Aug 18, 2024

[fix](stream load) do not throw exception but skip record when can not find database (#39360) #39527

Merged

[fix](stream load) do not throw exception but skip record when can not find database (#39360) #39531

Merged

dataroaring added the dev/2.0.15-merged label Aug 19, 2024

dataroaring removed the dev/2.0.x label Aug 19, 2024

dataroaring added dev/2.1.6-merged and removed dev/2.1.x labels Aug 19, 2024

gavinchou mentioned this pull request Oct 13, 2024

Release Note 3.0.2 #41558

Closed

Conversation

sollhui commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

sollhui commented Aug 14, 2024

Uh oh!

sollhui commented Aug 14, 2024

Uh oh!

sollhui commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 15, 2024

Uh oh!

github-actions bot commented Aug 15, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Jibing-Li commented Aug 15, 2024

Uh oh!

doris-robot commented Aug 15, 2024

Uh oh!

doris-robot commented Aug 15, 2024

Uh oh!

doris-robot commented Aug 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants