[fix](compaction) fix null pointer when retrieving CPU load average by luwei16 · Pull Request #37171 · apache/doris

luwei16 · 2024-07-02T12:59:40Z

1 When enable_system_metrics = false, system_metrics is a null pointer
2 When BE exits, system_metrics has been destructed, and the compaction background thread may still be running

doris-robot · 2024-07-02T12:59:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

luwei16 · 2024-07-02T12:59:55Z

run buildall

github-actions · 2024-07-02T13:05:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-02T13:29:27Z

TPC-H: Total hot run time: 39839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 664aba33881842adfca158313b4af9a0166c9c3b, data reload: false

------ Round 1 ----------------------------------
q1	17617	4575	4306	4306
q2	2014	199	194	194
q3	10490	1222	1075	1075
q4	10210	783	788	783
q5	7508	2692	2789	2692
q6	227	141	137	137
q7	973	606	614	606
q8	9507	2115	2064	2064
q9	8868	6486	6531	6486
q10	8959	3735	3749	3735
q11	446	243	237	237
q12	514	230	227	227
q13	18820	2973	2974	2973
q14	270	233	219	219
q15	517	492	487	487
q16	505	377	379	377
q17	952	685	626	626
q18	7999	7445	7479	7445
q19	4482	1386	1389	1386
q20	652	318	334	318
q21	4964	3142	3911	3142
q22	399	338	324	324
Total cold run time: 116893 ms
Total hot run time: 39839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4964	4251	4423	4251
q2	392	269	266	266
q3	2998	2725	2743	2725
q4	1889	1568	1591	1568
q5	5223	5285	5236	5236
q6	217	131	128	128
q7	2121	1747	1730	1730
q8	3186	3350	3311	3311
q9	8348	8332	8262	8262
q10	3881	3657	3649	3649
q11	595	495	480	480
q12	757	610	587	587
q13	16420	2972	2990	2972
q14	294	265	281	265
q15	527	488	471	471
q16	462	416	420	416
q17	1777	1491	1479	1479
q18	7557	7464	7320	7320
q19	1699	1525	1488	1488
q20	2014	1766	1770	1766
q21	4885	4736	4626	4626
q22	615	526	521	521
Total cold run time: 70821 ms
Total hot run time: 53517 ms

doris-robot · 2024-07-02T13:40:31Z

TPC-DS: Total hot run time: 172619 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 664aba33881842adfca158313b4af9a0166c9c3b, data reload: false

query1	919	369	372	369
query2	6458	2497	2214	2214
query3	6642	208	216	208
query4	18924	17768	17158	17158
query5	4234	483	472	472
query6	273	189	156	156
query7	4588	301	287	287
query8	308	268	268	268
query9	8431	2416	2386	2386
query10	620	302	281	281
query11	10531	10081	10032	10032
query12	132	87	82	82
query13	1648	368	364	364
query14	10131	7888	6905	6905
query15	242	183	187	183
query16	7955	276	284	276
query17	1875	569	546	546
query18	1886	279	275	275
query19	208	152	158	152
query20	93	81	87	81
query21	211	132	126	126
query22	4257	4224	3921	3921
query23	33879	33004	33037	33004
query24	11096	2875	2863	2863
query25	648	388	393	388
query26	1464	162	156	156
query27	2923	315	321	315
query28	7511	2084	2086	2084
query29	950	644	630	630
query30	286	151	159	151
query31	966	729	757	729
query32	100	55	57	55
query33	864	293	289	289
query34	941	459	468	459
query35	730	656	608	608
query36	1085	921	904	904
query37	143	76	100	76
query38	2927	2738	2761	2738
query39	843	798	799	798
query40	234	124	122	122
query41	53	50	51	50
query42	113	97	96	96
query43	568	530	553	530
query44	1249	717	713	713
query45	187	159	158	158
query46	1072	743	726	726
query47	1854	1799	1806	1799
query48	387	312	287	287
query49	1116	408	406	406
query50	770	376	380	376
query51	6892	6784	6788	6784
query52	103	90	94	90
query53	355	292	282	282
query54	934	442	445	442
query55	74	73	76	73
query56	280	284	268	268
query57	1128	1016	1047	1016
query58	260	235	238	235
query59	3388	3177	3064	3064
query60	298	275	283	275
query61	95	95	97	95
query62	664	426	477	426
query63	313	290	290	290
query64	9960	2273	1784	1784
query65	3438	3087	3122	3087
query66	1341	334	331	331
query67	15676	15052	15091	15052
query68	4707	539	560	539
query69	532	313	300	300
query70	1167	1153	1078	1078
query71	443	279	267	267
query72	8591	5512	5647	5512
query73	731	324	324	324
query74	5998	5459	5499	5459
query75	4347	2640	2648	2640
query76	3317	954	923	923
query77	665	348	300	300
query78	10416	9852	9763	9763
query79	2418	509	524	509
query80	2277	466	468	466
query81	560	223	219	219
query82	889	105	104	104
query83	304	169	166	166
query84	258	92	81	81
query85	1350	277	270	270
query86	439	339	325	325
query87	3269	3101	3095	3095
query88	3932	2338	2336	2336
query89	477	380	381	380
query90	2057	190	195	190
query91	127	101	101	101
query92	69	49	49	49
query93	1583	508	509	508
query94	1450	187	242	187
query95	402	328	316	316
query96	587	265	269	265
query97	3239	3049	3052	3049
query98	217	203	188	188
query99	1092	870	864	864
Total cold run time: 278634 ms
Total hot run time: 172619 ms

doris-robot · 2024-07-02T13:46:02Z

ClickBench: Total hot run time: 30.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 664aba33881842adfca158313b4af9a0166c9c3b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.51	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.13
query13	0.58	0.58	0.57
query14	0.81	0.79	0.79
query15	0.87	0.81	0.82
query16	0.35	0.37	0.35
query17	1.02	1.04	0.99
query18	0.22	0.28	0.24
query19	1.75	1.73	1.76
query20	0.01	0.01	0.01
query21	15.46	0.75	0.66
query22	4.66	7.22	1.84
query23	18.30	1.38	1.24
query24	2.14	0.22	0.23
query25	0.16	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.23	1.02	0.99
query29	12.70	3.35	3.32
query30	0.25	0.06	0.05
query31	2.86	0.39	0.38
query32	3.29	0.49	0.46
query33	2.91	2.89	2.90
query34	17.00	4.40	4.46
query35	4.50	4.53	4.56
query36	0.65	0.47	0.49
query37	0.19	0.15	0.15
query38	0.17	0.16	0.14
query39	0.05	0.03	0.04
query40	0.17	0.14	0.15
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.96 s
Total hot run time: 30.54 s

dataroaring

LGTM

github-actions · 2024-07-02T13:55:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-02T13:55:27Z

PR approved by anyone and no changes requested.

gavinchou · 2024-07-02T13:59:01Z


 double SystemMetrics::get_load_average_1_min() {
-    return _load_average_metrics->load_average_1_minutes->value();
+    if (!_load_average_metrics) {


it seems incorrect

luwei16 · 2024-07-03T02:34:27Z

run buildall

github-actions · 2024-07-03T02:39:57Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T03:11:51Z

TPC-H: Total hot run time: 39578 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e35319d2240398ea31b6c069aefd6592ec81e4c8, data reload: false

------ Round 1 ----------------------------------
q1	17613	4387	4258	4258
q2	2011	198	186	186
q3	10468	1126	1062	1062
q4	10188	839	779	779
q5	7549	2658	2620	2620
q6	231	140	142	140
q7	955	603	608	603
q8	9228	2063	2031	2031
q9	8618	6466	6449	6449
q10	9132	3746	3731	3731
q11	458	242	245	242
q12	490	240	247	240
q13	17779	2977	3023	2977
q14	277	228	230	228
q15	527	482	499	482
q16	503	391	383	383
q17	961	659	731	659
q18	7858	7294	7423	7294
q19	5712	1505	1386	1386
q20	661	323	327	323
q21	4906	3160	3923	3160
q22	407	345	353	345
Total cold run time: 116532 ms
Total hot run time: 39578 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4391	4246	4266	4246
q2	379	260	278	260
q3	2998	2765	2856	2765
q4	1966	1774	1656	1656
q5	5658	5532	5483	5483
q6	244	137	135	135
q7	2190	1856	1942	1856
q8	3233	3393	3433	3393
q9	8656	8640	8808	8640
q10	4106	3894	3758	3758
q11	578	501	491	491
q12	830	627	652	627
q13	16127	3191	3163	3163
q14	308	285	286	285
q15	528	504	487	487
q16	486	431	424	424
q17	1864	1589	1485	1485
q18	8177	7946	7796	7796
q19	1768	1686	1606	1606
q20	2123	1836	1870	1836
q21	8172	4910	4766	4766
q22	630	540	538	538
Total cold run time: 75412 ms
Total hot run time: 55696 ms

doris-robot · 2024-07-03T03:23:07Z

TPC-DS: Total hot run time: 173446 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e35319d2240398ea31b6c069aefd6592ec81e4c8, data reload: false

query1	914	385	372	372
query2	6439	2514	2341	2341
query3	6628	206	210	206
query4	19531	17397	17215	17215
query5	3678	476	470	470
query6	258	160	176	160
query7	4580	293	291	291
query8	305	320	311	311
query9	8512	2353	2339	2339
query10	584	302	272	272
query11	10537	10121	9992	9992
query12	112	87	87	87
query13	1649	376	382	376
query14	8695	7894	6567	6567
query15	230	187	196	187
query16	7735	276	275	275
query17	1896	564	537	537
query18	1956	279	275	275
query19	200	157	158	157
query20	89	82	84	82
query21	210	132	125	125
query22	4393	4128	4007	4007
query23	33821	33616	33554	33554
query24	10406	2949	2892	2892
query25	623	400	391	391
query26	712	157	159	157
query27	2232	331	332	331
query28	6032	2160	2112	2112
query29	894	689	645	645
query30	241	170	171	170
query31	964	786	765	765
query32	95	57	58	57
query33	655	303	305	303
query34	867	484	492	484
query35	766	648	675	648
query36	1133	1006	1014	1006
query37	150	77	86	77
query38	2916	2850	2843	2843
query39	873	831	832	831
query40	213	133	132	132
query41	60	55	54	54
query42	119	105	100	100
query43	593	565	532	532
query44	1066	751	734	734
query45	190	173	175	173
query46	1078	715	710	710
query47	1842	1765	1763	1763
query48	368	315	304	304
query49	874	496	410	410
query50	766	392	397	392
query51	6787	6889	6727	6727
query52	109	89	96	89
query53	354	295	288	288
query54	884	445	436	436
query55	75	85	72	72
query56	275	254	279	254
query57	1155	1055	1063	1055
query58	240	240	251	240
query59	3333	3410	3255	3255
query60	293	279	275	275
query61	109	96	92	92
query62	604	434	421	421
query63	318	294	294	294
query64	8537	2284	1773	1773
query65	3162	3100	3066	3066
query66	749	323	324	323
query67	15649	14980	14998	14980
query68	8638	548	563	548
query69	714	409	332	332
query70	1435	1135	1123	1123
query71	509	271	266	266
query72	8610	5275	5506	5275
query73	2212	323	323	323
query74	5972	5506	5571	5506
query75	5047	2626	2638	2626
query76	5226	928	911	911
query77	770	302	343	302
query78	10542	9747	9703	9703
query79	8256	511	504	504
query80	1047	481	465	465
query81	553	214	214	214
query82	489	108	108	108
query83	326	175	164	164
query84	276	85	87	85
query85	915	275	277	275
query86	347	336	316	316
query87	3303	3075	3101	3075
query88	4395	2451	2449	2449
query89	527	380	381	380
query90	2061	189	189	189
query91	124	100	97	97
query92	57	51	51	51
query93	6686	504	506	504
query94	1337	188	184	184
query95	397	306	313	306
query96	611	265	274	265
query97	3175	3002	3021	3002
query98	209	208	205	205
query99	1125	854	839	839
Total cold run time: 288204 ms
Total hot run time: 173446 ms

doris-robot · 2024-07-03T03:28:38Z

ClickBench: Total hot run time: 30.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e35319d2240398ea31b6c069aefd6592ec81e4c8, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.49	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.54	0.54
query11	0.15	0.11	0.11
query12	0.16	0.13	0.14
query13	0.59	0.58	0.58
query14	0.76	0.79	0.77
query15	0.85	0.81	0.81
query16	0.37	0.37	0.38
query17	0.99	1.05	1.04
query18	0.23	0.27	0.23
query19	1.90	1.69	1.75
query20	0.01	0.01	0.01
query21	15.44	0.76	0.66
query22	4.52	7.19	2.08
query23	18.29	1.38	1.24
query24	2.04	0.22	0.22
query25	0.15	0.09	0.09
query26	0.26	0.18	0.18
query27	0.09	0.08	0.09
query28	13.27	1.00	1.01
query29	12.66	3.24	3.23
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.47	0.46
query33	2.84	2.88	2.89
query34	17.10	4.36	4.42
query35	4.46	4.46	4.52
query36	0.67	0.45	0.48
query37	0.19	0.15	0.16
query38	0.18	0.14	0.16
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.81 s
Total hot run time: 30.64 s

dataroaring

LGTM

github-actions · 2024-07-03T14:46:40Z

PR approved by at least one committer and no changes requested.

…pache#37171) 1 When enable_system_metrics = false, system_metrics is a null pointer 2 When BE exits, system_metrics has been destructed, and the compaction background thread may still be running

…paction score and workload (#37491) adjust compaction concurrency based on compaction score and workload #36672 fix null pointer when retrieving CPU load average #37171

…37171) 1 When enable_system_metrics = false, system_metrics is a null pointer 2 When BE exits, system_metrics has been destructed, and the compaction background thread may still be running

[fix](compaction)fix null pointer when retrieving CPU load average

664aba3

luwei16 changed the title ~~[fix](compaction)fix null pointer when retrieving CPU load average~~ [fix](compaction) fix null pointer when retrieving CPU load average Jul 2, 2024

wm1581066 requested a review from gavinchou July 2, 2024 13:12

dataroaring previously approved these changes Jul 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024

github-actions bot added the reviewed label Jul 2, 2024

gavinchou reviewed Jul 2, 2024

View reviewed changes

fix

e35319d

luwei16 dismissed dataroaring’s stale review via e35319d July 3, 2024 02:33

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

dataroaring approved these changes Jul 3, 2024

View reviewed changes

Yukang-Lian approved these changes Jul 3, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

dataroaring merged commit aa038e9 into apache:master Jul 3, 2024

dataroaring added the dev/2.1.x label Jul 5, 2024

luwei16 mentioned this pull request Jul 8, 2024

[[enhancement](compaction) adjust compaction concurrency based on compaction score and workload #37491

Merged

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 9, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Conversation

luwei16 commented Jul 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

luwei16 commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

gavinchou Jul 2, 2024

Choose a reason for hiding this comment

Uh oh!

luwei16 Jul 3, 2024

Choose a reason for hiding this comment

Uh oh!

luwei16 commented Jul 3, 2024

Uh oh!

github-actions bot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

luwei16 commented Jul 2, 2024 •

edited

Loading