Skip to content

[opt](catalog) modify some meta cache logic#38506

Merged
morningman merged 2 commits intoapache:masterfrom
morningman:cache_fix
Aug 10, 2024
Merged

[opt](catalog) modify some meta cache logic#38506
morningman merged 2 commits intoapache:masterfrom
morningman:cache_fix

Conversation

@morningman
Copy link
Copy Markdown
Contributor

@morningman morningman commented Jul 30, 2024 •
edited
Loading

  1. Add a new FE config max_meta_object_cache_num to control the meta number of db/table in external catalog
  2. Shorten the expire time of some cache to make data more refresh

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 30, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 7, 2024
@morningman morningman changed the title [fix] cache [opt](catalog) modify some meta cache logic Aug 7, 2024
@morningman morningman marked this pull request as ready for review August 7, 2024 06:09
@morningman
Copy link
Copy Markdown
Contributor Author

morningman commented Aug 7, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 7, 2024

TPC-H: Total hot run time: 41715 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0485f5fa13af987ea9a186ca594ef575777a53a2, data reload: false

------ Round 1 ----------------------------------
q1	17673	4057	4063	4057
q2	2019	199	208	199
q3	10444	1339	1363	1339
q4	10166	854	919	854
q5	7608	3001	2965	2965
q6	220	138	138	138
q7	1062	628	613	613
q8	9455	1863	1945	1863
q9	8506	6581	6568	6568
q10	8785	3873	3867	3867
q11	434	245	242	242
q12	406	229	236	229
q13	17755	2944	2946	2944
q14	280	247	244	244
q15	518	486	475	475
q16	522	390	388	388
q17	968	945	917	917
q18	8080	7368	7288	7288
q19	1521	1219	1208	1208
q20	552	331	354	331
q21	5306	4738	4711	4711
q22	349	275	276	275
Total cold run time: 112629 ms
Total hot run time: 41715 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4051	4031	4001	4001
q2	330	231	223	223
q3	2989	2996	3179	2996
q4	1987	2035	1964	1964
q5	5591	5483	5477	5477
q6	222	129	129	129
q7	2120	1827	1829	1827
q8	3281	3344	3342	3342
q9	8638	8714	8768	8714
q10	3939	4002	3908	3908
q11	545	467	475	467
q12	819	593	588	588
q13	16349	3079	3116	3079
q14	316	288	266	266
q15	542	496	476	476
q16	462	424	421	421
q17	1765	1733	1738	1733
q18	8192	7811	7694	7694
q19	1763	1700	1702	1700
q20	2108	1836	1848	1836
q21	5692	5404	5265	5265
q22	524	477	466	466
Total cold run time: 72225 ms
Total hot run time: 56572 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 7, 2024

TPC-DS: Total hot run time: 169388 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0485f5fa13af987ea9a186ca594ef575777a53a2, data reload: false

query1	912	383	367	367
query2	6460	1715	1713	1713
query3	6650	210	222	210
query4	19640	17584	17322	17322
query5	3647	507	518	507
query6	289	188	172	172
query7	4596	297	314	297
query8	256	202	198	198
query9	8534	2326	2314	2314
query10	424	277	270	270
query11	10645	10110	10058	10058
query12	125	90	93	90
query13	1651	380	364	364
query14	8506	6995	7776	6995
query15	196	168	162	162
query16	6834	425	465	425
query17	935	585	540	540
query18	1309	278	279	278
query19	202	142	143	142
query20	93	92	83	83
query21	203	99	109	99
query22	4204	4162	3973	3973
query23	33796	33843	33760	33760
query24	9354	3124	3086	3086
query25	659	406	405	405
query26	1128	154	155	154
query27	2401	279	292	279
query28	6913	1996	1972	1972
query29	878	442	455	442
query30	243	152	149	149
query31	940	784	803	784
query32	102	53	54	53
query33	668	332	322	322
query34	919	503	502	502
query35	869	766	770	766
query36	1038	877	901	877
query37	154	81	82	81
query38	2912	2857	2782	2782
query39	873	816	841	816
query40	224	115	115	115
query41	44	49	45	45
query42	122	99	108	99
query43	469	417	440	417
query44	1194	723	714	714
query45	213	185	184	184
query46	1104	808	789	789
query47	1811	1737	1724	1724
query48	383	303	290	290
query49	904	417	434	417
query50	895	432	423	423
query51	6741	6713	6643	6643
query52	110	87	89	87
query53	250	182	188	182
query54	631	442	445	442
query55	77	73	72	72
query56	277	259	255	255
query57	1116	1044	1069	1044
query58	252	259	266	259
query59	2691	2340	2283	2283
query60	311	268	274	268
query61	101	94	95	94
query62	856	655	650	650
query63	219	181	175	175
query64	4519	1904	1862	1862
query65	3150	3086	3129	3086
query66	831	324	330	324
query67	15110	14763	14921	14763
query68	4320	563	572	563
query69	444	302	296	296
query70	1060	1047	1056	1047
query71	379	275	274	274
query72	7066	2661	2465	2465
query73	759	332	327	327
query74	6038	5718	5622	5622
query75	3330	2718	2691	2691
query76	2186	1219	1290	1219
query77	430	304	307	304
query78	9316	8860	8956	8860
query79	2422	527	521	521
query80	1200	501	500	500
query81	571	220	226	220
query82	1043	130	135	130
query83	235	169	166	166
query84	257	79	83	79
query85	1160	312	307	307
query86	455	282	289	282
query87	3273	3109	3066	3066
query88	3018	2372	2392	2372
query89	382	298	287	287
query90	1682	190	188	188
query91	121	100	160	100
query92	60	48	49	48
query93	1952	612	621	612
query94	804	306	295	295
query95	379	257	269	257
query96	604	279	280	279
query97	3226	3035	3092	3035
query98	225	211	192	192
query99	1617	1269	1279	1269
Total cold run time: 254777 ms
Total hot run time: 169388 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 7, 2024

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0485f5fa13af987ea9a186ca594ef575777a53a2, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.06
query5	0.47	0.48	0.48
query6	1.14	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.50
query10	0.56	0.57	0.57
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.61	0.62	0.60
query14	0.77	0.81	0.77
query15	0.92	0.86	0.87
query16	0.36	0.36	0.35
query17	0.97	1.00	1.00
query18	0.21	0.21	0.21
query19	1.81	1.76	1.73
query20	0.02	0.01	0.01
query21	15.40	0.75	0.64
query22	3.59	5.33	1.69
query23	17.55	1.36	1.24
query24	2.25	0.24	0.21
query25	0.19	0.08	0.08
query26	0.32	0.23	0.22
query27	0.47	0.23	0.23
query28	13.19	1.01	0.96
query29	12.52	3.30	3.29
query30	0.25	0.06	0.05
query31	2.88	0.41	0.40
query32	3.22	0.51	0.48
query33	2.96	2.91	2.98
query34	15.45	4.27	4.25
query35	4.27	4.28	4.31
query36	0.67	0.48	0.48
query37	0.20	0.16	0.17
query38	0.17	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.76 s
Total hot run time: 30.26 s

@morningman
Copy link
Copy Markdown
Contributor Author

morningman commented Aug 8, 2024

run buildall

Jibing-Li
Jibing-Li approved these changes Aug 8, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@Jibing-Li Jibing-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 8, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 8, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 8, 2024

TPC-H: Total hot run time: 39261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e64e381520c8c55529bbf1cfc4c98a42dbe9e7ce, data reload: false

------ Round 1 ----------------------------------
q1	17627	4309	4301	4301
q2	2020	177	193	177
q3	10469	1169	1138	1138
q4	10141	781	700	700
q5	7508	2498	2464	2464
q6	220	138	138	138
q7	952	586	586	586
q8	9212	1904	1901	1901
q9	8819	6541	6554	6541
q10	7112	2237	2170	2170
q11	449	237	243	237
q12	482	216	211	211
q13	18766	2967	2974	2967
q14	282	229	233	229
q15	523	478	476	476
q16	490	384	381	381
q17	954	662	682	662
q18	7948	7381	7530	7381
q19	6632	951	1101	951
q20	688	333	339	333
q21	5192	4634	4317	4317
q22	1108	1007	1000	1000
Total cold run time: 117594 ms
Total hot run time: 39261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4463	4226	4227	4226
q2	377	269	270	269
q3	2848	2625	2707	2625
q4	1952	1743	1715	1715
q5	5619	5475	5465	5465
q6	216	129	134	129
q7	2146	1801	1849	1801
q8	3312	3505	3530	3505
q9	8793	8780	8839	8780
q10	3462	3342	3221	3221
q11	591	495	498	495
q12	821	624	654	624
q13	16105	3197	3172	3172
q14	323	296	288	288
q15	525	486	500	486
q16	509	451	444	444
q17	1776	1535	1537	1535
q18	7929	7966	7873	7873
q19	1752	1621	1588	1588
q20	2167	1886	1882	1882
q21	5544	5369	5083	5083
q22	1125	1040	1050	1040
Total cold run time: 72355 ms
Total hot run time: 56246 ms

@morningman
Copy link
Copy Markdown
Contributor Author

morningman commented Aug 9, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

TPC-H: Total hot run time: 39244 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3d439e5bce3ffe58080289dbecbffbf534e5fb5, data reload: false

------ Round 1 ----------------------------------
q1	17632	4380	4316	4316
q2	2014	176	172	172
q3	10484	1215	1074	1074
q4	10132	768	671	671
q5	7499	2538	2503	2503
q6	231	139	140	139
q7	997	596	593	593
q8	9226	1898	1915	1898
q9	8614	6538	6573	6538
q10	7050	2182	2110	2110
q11	463	250	251	250
q12	475	232	225	225
q13	17767	3016	2985	2985
q14	295	233	233	233
q15	528	486	501	486
q16	506	391	381	381
q17	972	661	753	661
q18	8049	7418	7283	7283
q19	5551	1070	953	953
q20	702	334	343	334
q21	5328	4445	4591	4445
q22	1065	994	999	994
Total cold run time: 115580 ms
Total hot run time: 39244 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4496	4315	4225	4225
q2	366	271	265	265
q3	2828	2611	2681	2611
q4	1925	1714	1718	1714
q5	5504	5581	5501	5501
q6	223	139	136	136
q7	2113	1745	1781	1745
q8	3339	3458	3415	3415
q9	8771	8676	8912	8676
q10	3489	3262	3299	3262
q11	621	481	489	481
q12	797	607	603	603
q13	15875	3196	3208	3196
q14	324	286	315	286
q15	544	492	494	492
q16	492	439	450	439
q17	1806	1571	1520	1520
q18	8137	7951	7986	7951
q19	1806	1575	1701	1575
q20	2116	1869	1900	1869
q21	9724	5418	5463	5418
q22	1065	1036	997	997
Total cold run time: 76361 ms
Total hot run time: 56377 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

TPC-DS: Total hot run time: 202407 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3d439e5bce3ffe58080289dbecbffbf534e5fb5, data reload: false

query1	909	381	368	368
query2	6466	1913	1750	1750
query3	6640	209	213	209
query4	33130	23122	23107	23107
query5	3607	491	501	491
query6	266	183	167	167
query7	4571	300	289	289
query8	244	200	199	199
query9	8610	2471	2438	2438
query10	539	438	428	428
query11	16306	15009	14957	14957
query12	132	101	99	99
query13	1660	376	365	365
query14	9394	7747	7667	7667
query15	251	205	215	205
query16	7591	508	432	432
query17	1713	552	554	552
query18	1820	271	276	271
query19	211	144	135	135
query20	113	106	101	101
query21	209	109	103	103
query22	4472	4480	4499	4480
query23	34344	33661	33713	33661
query24	10808	2737	2668	2668
query25	601	378	367	367
query26	698	164	162	162
query27	2277	288	292	288
query28	6237	2037	2033	2033
query29	765	411	406	406
query30	253	146	143	143
query31	951	701	751	701
query32	92	53	57	53
query33	727	290	276	276
query34	904	458	469	458
query35	954	824	787	787
query36	1063	904	944	904
query37	138	85	83	83
query38	4169	4193	4154	4154
query39	1449	1376	1395	1376
query40	209	118	123	118
query41	48	48	82	48
query42	116	94	96	94
query43	505	459	449	449
query44	1216	721	724	721
query45	239	201	205	201
query46	1083	750	735	735
query47	1829	1752	1747	1747
query48	362	290	291	290
query49	835	412	414	412
query50	797	406	406	406
query51	6801	6705	6581	6581
query52	107	89	87	87
query53	256	182	183	182
query54	870	454	449	449
query55	75	74	74	74
query56	257	244	239	239
query57	1126	1068	1072	1068
query58	218	225	227	225
query59	2961	2668	2688	2668
query60	279	269	258	258
query61	98	130	93	93
query62	781	646	633	633
query63	215	180	180	180
query64	9221	2253	1727	1727
query65	3221	3127	3148	3127
query66	755	337	344	337
query67	15404	14872	14802	14802
query68	4466	538	538	538
query69	404	380	361	361
query70	1190	1148	1099	1099
query71	430	282	273	273
query72	18329	17245	17084	17084
query73	771	327	329	327
query74	9047	8699	8802	8699
query75	3360	2691	2660	2660
query76	2384	989	990	989
query77	507	313	293	293
query78	9683	9128	9774	9128
query79	2970	511	510	510
query80	2364	494	491	491
query81	616	228	234	228
query82	777	144	136	136
query83	314	143	145	143
query84	267	80	75	75
query85	1935	292	275	275
query86	313	300	302	300
query87	4612	4525	4561	4525
query88	3897	2488	2513	2488
query89	411	282	285	282
query90	1768	194	193	193
query91	122	91	100	91
query92	62	48	48	48
query93	2617	541	539	539
query94	687	301	341	301
query95	365	271	264	264
query96	600	289	272	272
query97	3235	3066	3055	3055
query98	219	202	195	195
query99	1594	1269	1261	1261
Total cold run time: 305484 ms
Total hot run time: 202407 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Aug 9, 2024

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3d439e5bce3ffe58080289dbecbffbf534e5fb5, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.07
query5	0.50	0.50	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.49	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	0.78	0.79	0.78
query15	0.86	0.83	0.82
query16	0.35	0.36	0.35
query17	1.05	1.00	0.99
query18	0.23	0.22	0.21
query19	1.79	1.80	1.77
query20	0.01	0.01	0.01
query21	15.38	0.76	0.66
query22	4.50	6.95	1.68
query23	18.29	1.44	1.28
query24	2.18	0.22	0.23
query25	0.16	0.08	0.08
query26	0.29	0.22	0.21
query27	0.46	0.24	0.23
query28	13.18	1.02	1.02
query29	12.62	3.27	3.25
query30	0.24	0.06	0.04
query31	2.91	0.40	0.40
query32	3.26	0.50	0.47
query33	2.93	2.95	2.98
query34	17.13	4.35	4.33
query35	4.42	4.39	4.38
query36	0.65	0.50	0.46
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 110.14 s
Total hot run time: 30.48 s

wuwenchi
wuwenchi approved these changes Aug 10, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@wuwenchi wuwenchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 289100b into apache:master Aug 10, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@morningman @dataroaring
[opt](catalog) modify some meta cache logic (#38506)
6d1f682 
1. Add a new FE config `max_meta_object_cache_num` to control the meta
number of db/table in external catalog
2. Shorten the expire time of some cache to make data more refresh
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@morningman @wyxxxcat
[opt](catalog) modify some meta cache logic (apache#38506)
6cd69b2 
1. Add a new FE config `max_meta_object_cache_num` to control the meta
number of db/table in external catalog
2. Shorten the expire time of some cache to make data more refresh
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@morningman @dataroaring
[opt](catalog) modify some meta cache logic (#38506)
0d7d2c5 
1. Add a new FE config `max_meta_object_cache_num` to control the meta
number of db/table in external catalog
2. Shorten the expire time of some cache to make data more refresh
morningman added a commit to morningman/doris that referenced this pull request Aug 20, 2024
@morningman
[opt](catalog) modify some meta cache logic (apache#38506)
274289d 
1. Add a new FE config `max_meta_object_cache_num` to control the meta
number of db/table in external catalog
2. Shorten the expire time of some cache to make data more refresh
@morningman morningman mentioned this pull request Aug 20, 2024
Merged
morningman added a commit that referenced this pull request Aug 20, 2024
@morningman
[opt](catalog) modify some meta cache logic (#38506) (#39628)
0e21dba 
#38506
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wuwenchi wuwenchi wuwenchi approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @doris-robot @wuwenchi @Jibing-Li @dataroaring