[Fix](cloud) CloudStorageEngine should wait compaction task done when stoping it by hust-hhb · Pull Request #39302 · apache/doris

hust-hhb · 2024-08-13T09:27:18Z

When stoping doris, CloudStorageEngine should wait compaction task done, like StorageEngine does, if we don't wait, the compaction task run in background may core beacause these task will access CloudStorageEngine to get _txn_delete_bitmap_cache

… stoping it

doris-robot · 2024-08-13T09:27:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hust-hhb · 2024-08-13T09:27:32Z

run buildall

github-actions · 2024-08-13T09:34:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-13T10:08:25Z

TPC-H: Total hot run time: 39352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65adfda85861f126b98ae538dfc9921c274b3b56, data reload: false

------ Round 1 ----------------------------------
q1	17617	4346	4230	4230
q2	2020	181	179	179
q3	11735	983	1162	983
q4	10520	783	807	783
q5	7744	2829	2823	2823
q6	220	133	137	133
q7	950	595	587	587
q8	9532	2035	2071	2035
q9	9198	6535	6512	6512
q10	7025	2216	2196	2196
q11	501	240	251	240
q12	388	228	214	214
q13	17847	2991	2954	2954
q14	278	242	245	242
q15	522	488	488	488
q16	501	401	385	385
q17	961	694	722	694
q18	8151	7387	7424	7387
q19	7338	1112	952	952
q20	712	328	328	328
q21	5320	4015	4347	4015
q22	1069	992	1016	992
Total cold run time: 120149 ms
Total hot run time: 39352 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4520	4247	4220	4220
q2	385	274	272	272
q3	2863	2736	2760	2736
q4	1918	1692	1669	1669
q5	5627	5606	5500	5500
q6	231	139	139	139
q7	2211	1746	1764	1746
q8	3268	3445	3403	3403
q9	8741	8858	8713	8713
q10	3490	3191	3248	3191
q11	596	523	520	520
q12	824	665	635	635
q13	17079	3201	3202	3201
q14	336	321	290	290
q15	539	494	484	484
q16	497	453	446	446
q17	1809	1522	1521	1521
q18	8146	8145	7803	7803
q19	1830	1579	1556	1556
q20	2972	1899	1860	1860
q21	5914	5203	5234	5203
q22	1142	1039	1075	1039
Total cold run time: 74938 ms
Total hot run time: 56147 ms

doris-robot · 2024-08-13T10:20:38Z

TPC-DS: Total hot run time: 189216 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 65adfda85861f126b98ae538dfc9921c274b3b56, data reload: false

query1	1264	879	850	850
query2	6338	1924	1862	1862
query3	10620	3730	3834	3730
query4	58775	25270	23453	23453
query5	5581	502	470	470
query6	439	159	166	159
query7	5890	287	284	284
query8	280	211	198	198
query9	8755	2439	2437	2437
query10	498	263	254	254
query11	18309	15001	15065	15001
query12	163	109	104	104
query13	1564	375	354	354
query14	11082	7763	7075	7075
query15	245	167	171	167
query16	7619	506	481	481
query17	1143	598	566	566
query18	1919	291	295	291
query19	286	144	152	144
query20	118	106	115	106
query21	206	102	110	102
query22	4759	4183	4275	4183
query23	34478	33284	33246	33246
query24	5885	2878	2864	2864
query25	527	369	379	369
query26	693	151	152	151
query27	1797	268	269	268
query28	3739	2053	2024	2024
query29	665	397	390	390
query30	225	143	150	143
query31	926	708	706	706
query32	81	54	52	52
query33	432	292	283	283
query34	852	458	470	458
query35	804	710	703	703
query36	1066	901	924	901
query37	130	78	80	78
query38	3838	3780	3812	3780
query39	1423	1408	1394	1394
query40	202	115	111	111
query41	45	44	44	44
query42	113	98	98	98
query43	492	460	445	445
query44	1061	729	733	729
query45	203	163	165	163
query46	1085	737	746	737
query47	1886	1833	1776	1776
query48	359	283	292	283
query49	736	408	411	408
query50	804	400	391	391
query51	6881	6694	6656	6656
query52	105	89	87	87
query53	248	184	177	177
query54	552	452	431	431
query55	76	73	71	71
query56	260	252	240	240
query57	1105	1056	1041	1041
query58	221	218	214	214
query59	2946	2625	2629	2625
query60	323	270	257	257
query61	102	109	95	95
query62	735	642	630	630
query63	208	177	178	177
query64	3318	1732	1700	1700
query65	3200	3141	3143	3141
query66	648	324	323	323
query67	15147	14945	14827	14827
query68	4511	543	534	534
query69	412	271	269	269
query70	1155	1066	1082	1066
query71	367	268	274	268
query72	7116	2289	2056	2056
query73	770	328	321	321
query74	9080	8702	8779	8702
query75	3339	2750	2741	2741
query76	2186	1002	924	924
query77	528	300	304	300
query78	9518	9804	8901	8901
query79	2175	520	536	520
query80	1275	479	487	479
query81	546	221	222	221
query82	528	136	133	133
query83	246	143	141	141
query84	263	73	75	73
query85	1157	281	274	274
query86	440	302	310	302
query87	4409	4184	4270	4184
query88	4228	2395	2411	2395
query89	391	285	281	281
query90	1925	193	209	193
query91	119	95	93	93
query92	57	48	47	47
query93	2354	528	520	520
query94	889	304	283	283
query95	357	270	265	265
query96	605	288	276	276
query97	3251	3035	3006	3006
query98	220	207	282	207
query99	1534	1283	1288	1283
Total cold run time: 312426 ms
Total hot run time: 189216 ms

doris-robot · 2024-08-13T10:26:10Z

ClickBench: Total hot run time: 31.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 65adfda85861f126b98ae538dfc9921c274b3b56, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.49	0.49
query6	1.13	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.47	0.48
query10	0.54	0.53	0.54
query11	0.14	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.58
query14	0.75	0.80	0.77
query15	0.84	0.82	0.82
query16	0.37	0.37	0.37
query17	1.03	0.99	1.06
query18	0.23	0.21	0.21
query19	1.87	1.80	1.80
query20	0.03	0.01	0.01
query21	15.38	0.74	0.66
query22	4.59	6.77	2.23
query23	18.33	1.44	1.22
query24	2.07	0.22	0.22
query25	0.14	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.31	1.01	1.01
query29	12.66	3.28	3.31
query30	0.24	0.05	0.05
query31	2.88	0.40	0.39
query32	3.28	0.49	0.47
query33	2.92	2.94	2.99
query34	17.00	4.32	4.35
query35	4.42	4.47	4.42
query36	0.65	0.48	0.49
query37	0.19	0.15	0.16
query38	0.16	0.14	0.16
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.13 s
Total hot run time: 31.02 s

github-actions · 2024-08-13T16:00:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-13T16:00:55Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

… stoping it (apache#39302) When stoping doris, CloudStorageEngine should wait compaction task done, like StorageEngine does, if we don't wait, the compaction task run in background may core beacause these task will access CloudStorageEngine to get _txn_delete_bitmap_cache

… stoping it (#39302) When stoping doris, CloudStorageEngine should wait compaction task done, like StorageEngine does, if we don't wait, the compaction task run in background may core beacause these task will access CloudStorageEngine to get _txn_delete_bitmap_cache

… stoping it (apache#39302) When stoping doris, CloudStorageEngine should wait compaction task done, like StorageEngine does, if we don't wait, the compaction task run in background may core beacause these task will access CloudStorageEngine to get _txn_delete_bitmap_cache

[Fix](cloud) CloudStorageEngine should wait compaction task done when…

65adfda

… stoping it

github-actions Bot added the doing label Aug 13, 2024

gavinchou approved these changes Aug 13, 2024

View reviewed changes

github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 13, 2024

dataroaring approved these changes Aug 14, 2024

View reviewed changes

dataroaring merged commit 181c202 into apache:master Aug 14, 2024

dataroaring added the dev/3.0.x label Aug 14, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

gavinchou mentioned this pull request Oct 13, 2024

Release Note 3.0.2 #41558

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](cloud) CloudStorageEngine should wait compaction task done when stoping it#39302

[Fix](cloud) CloudStorageEngine should wait compaction task done when stoping it#39302
dataroaring merged 1 commit intoapache:masterfrom
hust-hhb:wait-compaction-done

hust-hhb commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

hust-hhb commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

dataroaring left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

hust-hhb commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

hust-hhb commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

github-actions Bot commented Aug 13, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants