Skip to content

[opt](mysql serde) Avoid core dump when converting invalid block to mysql result #28069#36745

Merged
xiaokang merged 1 commit intoapache:branch-2.0from
zhiqiang-hhhh:pick_28069_to_upstream_branch-2.0
Jul 31, 2024
Merged

[opt](mysql serde) Avoid core dump when converting invalid block to mysql result #28069#36745
xiaokang merged 1 commit intoapache:branch-2.0from
zhiqiang-hhhh:pick_28069_to_upstream_branch-2.0

Conversation

@zhiqiang-hhhh
Copy link
Copy Markdown
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Jun 24, 2024

cherry pick from #28069

@zhiqiang-hhhh
[opt](mysql serde) Avoid core dump when converting invalid block to m…
0ab9b10 
…ysql result (apache#28069)

BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in apache#28030, so we add check branch to avoid BE core dump due to out of range related problem.
@zhiqiang-hhhh
Copy link
Copy Markdown
Contributor Author

zhiqiang-hhhh commented Jun 24, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 24, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

TeamCity be ut coverage result:
Function Coverage: 37.87% (8109/21415)
Line Coverage: 29.54% (66378/224742)
Region Coverage: 29.01% (34213/117942)
Branch Coverage: 24.88% (17565/70612)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0ab9b10834320c9aa057998b6f2fcea65eb0d812_0ab9b10834320c9aa057998b6f2fcea65eb0d812/report/index.html

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

TPC-H: Total hot run time: 49481 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ab9b10834320c9aa057998b6f2fcea65eb0d812, data reload: false

------ Round 1 ----------------------------------
q1	18027	4379	4286	4286
q2	2065	154	153	153
q3	10474	1897	1953	1897
q4	10333	1229	1322	1229
q5	8829	3852	3877	3852
q6	263	122	141	122
q7	2057	1597	1608	1597
q8	9270	2714	2705	2705
q9	10536	10161	10236	10161
q10	8661	3531	3501	3501
q11	421	237	252	237
q12	475	306	302	302
q13	18386	3923	4007	3923
q14	362	340	330	330
q15	503	451	455	451
q16	680	580	574	574
q17	1126	951	977	951
q18	7262	6764	6860	6764
q19	1795	1667	1613	1613
q20	530	318	302	302
q21	4434	4092	4088	4088
q22	539	443	455	443
Total cold run time: 117028 ms
Total hot run time: 49481 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4336	4294	4327	4294
q2	323	230	214	214
q3	4190	4108	4135	4108
q4	2774	2738	2747	2738
q5	7212	7164	7134	7134
q6	241	122	120	120
q7	3186	2800	2847	2800
q8	4343	4485	4478	4478
q9	16966	16666	16699	16666
q10	4245	4251	4294	4251
q11	768	688	671	671
q12	1024	853	855	853
q13	7928	3737	3755	3737
q14	456	427	425	425
q15	501	467	460	460
q16	757	674	674	674
q17	3808	3833	3858	3833
q18	8875	8651	8725	8651
q19	1718	1732	1590	1590
q20	2417	2133	2114	2114
q21	8472	8477	8526	8477
q22	1033	947	989	947
Total cold run time: 85573 ms
Total hot run time: 79235 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

TPC-DS: Total hot run time: 203847 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ab9b10834320c9aa057998b6f2fcea65eb0d812, data reload: false

query1	952	422	375	375
query2	6554	3018	2905	2905
query3	6923	205	204	204
query4	21366	18024	17987	17987
query5	19724	6480	6509	6480
query6	303	221	228	221
query7	4156	300	308	300
query8	426	376	373	373
query9	3077	2647	2581	2581
query10	413	297	282	282
query11	11417	10759	10796	10759
query12	121	75	74	74
query13	5603	670	701	670
query14	17880	13553	13221	13221
query15	368	247	250	247
query16	6450	302	265	265
query17	1716	1458	876	876
query18	2307	419	419	419
query19	198	146	149	146
query20	83	76	78	76
query21	186	96	93	93
query22	5353	5262	5248	5248
query23	32427	32051	32166	32051
query24	6952	6787	6541	6541
query25	538	440	416	416
query26	530	161	160	160
query27	1870	291	294	291
query28	6169	2358	2310	2310
query29	2880	2798	2866	2798
query30	240	165	167	165
query31	918	754	794	754
query32	66	62	63	62
query33	394	261	263	261
query34	846	474	475	474
query35	1117	924	938	924
query36	1380	1136	1061	1061
query37	87	59	59	59
query38	3058	2912	2949	2912
query39	1387	1326	1322	1322
query40	205	84	93	84
query41	45	43	42	42
query42	87	79	80	79
query43	664	737	691	691
query44	1116	717	724	717
query45	247	251	235	235
query46	1223	954	983	954
query47	1893	1951	1749	1749
query48	1038	713	714	713
query49	622	371	364	364
query50	855	609	624	609
query51	4784	4683	4606	4606
query52	91	83	83	83
query53	447	322	319	319
query54	2645	2465	2496	2465
query55	84	80	81	80
query56	242	215	208	208
query57	1181	1157	1138	1138
query58	220	214	190	190
query59	4341	3782	4098	3782
query60	214	206	209	206
query61	98	98	99	98
query62	877	478	454	454
query63	486	338	341	338
query64	2577	1537	1529	1529
query65	3606	3604	3601	3601
query66	777	388	378	378
query67	18247	15307	16444	15307
query68	8866	640	666	640
query69	570	362	360	360
query70	1733	1408	1317	1317
query71	423	311	307	307
query72	6606	3489	3503	3489
query73	740	317	316	316
query74	6337	5836	5914	5836
query75	5094	3646	3629	3629
query76	5020	1161	1230	1161
query77	748	250	252	250
query78	12385	11608	12013	11608
query79	7088	615	625	615
query80	1070	405	413	405
query81	510	234	233	233
query82	1530	97	102	97
query83	165	136	134	134
query84	269	69	70	69
query85	879	323	324	323
query86	331	303	288	288
query87	3276	3081	3016	3016
query88	5052	2263	2268	2263
query89	353	287	267	267
query90	2083	210	212	210
query91	175	138	150	138
query92	61	53	55	53
query93	4887	537	615	537
query94	771	211	217	211
query95	1138	1063	1042	1042
query96	641	326	319	319
query97	6498	6376	6355	6355
query98	188	176	180	176
query99	3033	852	801	801
Total cold run time: 313867 ms
Total hot run time: 203847 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

ClickBench: Total hot run time: 31.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ab9b10834320c9aa057998b6f2fcea65eb0d812, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.78	0.06	0.06
query5	0.53	0.52	0.52
query6	1.22	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.50	0.48
query10	0.55	0.54	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.80	0.76
query15	0.78	0.77	0.76
query16	0.36	0.39	0.39
query17	1.04	1.02	1.01
query18	0.24	0.23	0.26
query19	1.96	1.87	1.87
query20	0.02	0.01	0.01
query21	15.48	0.59	0.56
query22	1.99	2.50	1.74
query23	17.08	1.13	1.16
query24	5.94	1.82	0.90
query25	0.35	0.12	0.04
query26	0.70	0.15	0.14
query27	0.05	0.04	0.05
query28	6.75	0.81	0.74
query29	12.69	2.32	2.32
query30	0.61	0.54	0.53
query31	2.81	0.40	0.37
query32	3.37	0.50	0.50
query33	3.06	3.06	3.05
query34	15.25	4.82	4.79
query35	4.85	4.84	4.85
query36	1.07	1.02	1.02
query37	0.06	0.05	0.04
query38	0.03	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.48 s
Total hot run time: 31.09 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jun 24, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0ab9b10834320c9aa057998b6f2fcea65eb0d812 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@xiaokang xiaokang merged commit cfa1e64 into apache:branch-2.0 Jul 31, 2024
@zhiqiang-hhhh zhiqiang-hhhh deleted the pick_28069_to_upstream_branch-2.0 branch August 1, 2024 02:43
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@zhiqiang-hhhh
[opt](mysql serde) Avoid core dump when converting invalid block to m…
ec71134 
…ysql result apache#28069 (apache#36745)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiqiang-hhhh @doris-robot @xiaokang