[fix](table_options) fix potential NPE when quering table_options sys table by morningman · Pull Request #40944 · apache/doris

morningman · 2024-09-18T14:15:36Z

Fix unstable issue like:

2024-09-17 12:41:04,755 WARN (thrift-server-pool-4|384) [FrontendServiceImpl.fetchSchemaTableData():2292] Failed to fetchSchemaTableData
java.lang.NullPointerException: Cannot invoke "org.apache.doris.catalog.MaterializedIndexMeta.getSchema()" because the return value of "java.util.Map.get(Object)" is null
        at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:954) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:947) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.getBaseSchema(OlapTable.java:2097) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.getKeyColAsString(OlapTable.java:2167) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsForInternalCatalog(MetadataGenerator.java:1099) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsMetadataResult(MetadataGenerator.java:1189) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.tablefunction.MetadataGenerator.getSchemaTableData(MetadataGenerator.java:273) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2289) ~[doris-fe.jar:1.2-SNAPSHOT]
        at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at jdk.proxy2.$Proxy44.fetchSchemaTableData(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4572) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4552) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]

… table

doris-robot · 2024-09-18T14:15:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman · 2024-09-18T14:15:50Z

run buildall

doris-robot · 2024-09-18T14:50:29Z

TPC-H: Total hot run time: 41048 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca55c3b914925959e8a3a4f85054e0ef0793f078, data reload: false

------ Round 1 ----------------------------------
q1	17591	7312	7240	7240
q2	2034	160	157	157
q3	10609	1088	1139	1088
q4	10551	734	731	731
q5	7743	3052	2998	2998
q6	231	149	144	144
q7	983	613	603	603
q8	9414	1994	2009	1994
q9	6809	6382	6358	6358
q10	7037	2282	2284	2282
q11	451	240	243	240
q12	402	214	210	210
q13	17778	3008	2959	2959
q14	242	217	211	211
q15	570	521	520	520
q16	675	623	599	599
q17	969	779	817	779
q18	7247	6696	6680	6680
q19	1399	1029	950	950
q20	578	283	270	270
q21	3973	3098	3034	3034
q22	1131	1003	1001	1001
Total cold run time: 108417 ms
Total hot run time: 41048 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7242	7166	7196	7166
q2	324	226	223	223
q3	2902	2921	2997	2921
q4	2072	1752	1804	1752
q5	5588	5596	5531	5531
q6	220	143	136	136
q7	2245	1827	1760	1760
q8	3269	3424	3366	3366
q9	8702	8880	8676	8676
q10	3497	3487	3420	3420
q11	588	482	491	482
q12	788	593	609	593
q13	10085	3227	3169	3169
q14	313	284	280	280
q15	584	530	527	527
q16	714	664	680	664
q17	1817	1598	1602	1598
q18	8238	7626	7752	7626
q19	1731	1681	1513	1513
q20	2123	1910	1868	1868
q21	5447	5200	5368	5200
q22	1155	1063	1029	1029
Total cold run time: 69644 ms
Total hot run time: 59500 ms

doris-robot · 2024-09-18T15:02:57Z

TPC-DS: Total hot run time: 198937 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca55c3b914925959e8a3a4f85054e0ef0793f078, data reload: false

query1	1294	879	877	877
query2	6395	2051	2046	2046
query3	10804	3958	4048	3958
query4	63063	28978	23276	23276
query5	5116	478	465	465
query6	398	159	161	159
query7	5436	308	287	287
query8	311	227	219	219
query9	8311	2643	2637	2637
query10	450	301	265	265
query11	17187	15147	15662	15147
query12	158	100	101	100
query13	1475	443	409	409
query14	10420	7186	7018	7018
query15	208	175	181	175
query16	6795	492	515	492
query17	1153	625	600	600
query18	1510	320	315	315
query19	199	147	151	147
query20	128	115	106	106
query21	213	108	108	108
query22	4572	4529	4754	4529
query23	34760	34070	33961	33961
query24	5992	2857	2871	2857
query25	524	413	406	406
query26	637	159	158	158
query27	1645	281	285	281
query28	4321	2427	2403	2403
query29	664	424	429	424
query30	233	161	157	157
query31	961	790	802	790
query32	68	57	54	54
query33	454	286	286	286
query34	903	490	480	480
query35	860	708	723	708
query36	1067	933	933	933
query37	147	83	88	83
query38	3931	3906	3937	3906
query39	1489	1426	1393	1393
query40	215	95	96	95
query41	50	48	47	47
query42	117	95	94	94
query43	533	500	490	490
query44	1145	815	802	802
query45	191	164	166	164
query46	1125	756	757	756
query47	1925	1772	1823	1772
query48	483	369	361	361
query49	696	398	402	398
query50	837	415	403	403
query51	7102	6939	6958	6939
query52	96	82	88	82
query53	245	175	175	175
query54	567	445	459	445
query55	77	73	74	73
query56	266	238	247	238
query57	1228	1054	1077	1054
query58	215	228	233	228
query59	3215	2975	3032	2975
query60	284	254	263	254
query61	101	100	102	100
query62	780	680	672	672
query63	215	187	181	181
query64	1380	643	656	643
query65	3228	3184	3192	3184
query66	690	294	297	294
query67	16133	15750	15617	15617
query68	1270	549	569	549
query69	422	292	283	283
query70	1186	1130	1128	1128
query71	321	265	270	265
query72	6112	3963	4026	3963
query73	753	321	329	321
query74	9225	9032	8959	8959
query75	3348	2670	2673	2670
query76	1381	904	891	891
query77	451	298	290	290
query78	10121	9619	9682	9619
query79	1178	889	861	861
query80	850	567	560	560
query81	535	253	257	253
query82	1157	235	226	226
query83	247	153	153	153
query84	284	102	99	99
query85	750	368	354	354
query86	331	324	303	303
query87	4417	4361	4395	4361
query88	4340	4080	4057	4057
query89	385	358	358	358
query90	1575	307	306	306
query91	161	154	166	154
query92	73	70	71	70
query93	943	896	898	896
query94	638	383	362	362
query95	440	403	401	401
query96	488	483	487	483
query97	3216	3119	3098	3098
query98	224	229	222	222
query99	1443	1282	1271	1271
Total cold run time: 305656 ms
Total hot run time: 198937 ms

doris-robot · 2024-09-18T15:08:27Z

ClickBench: Total hot run time: 32.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca55c3b914925959e8a3a4f85054e0ef0793f078, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.09
query5	0.54	0.50	0.51
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.50
query10	0.56	0.54	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.97	2.98	3.11
query15	0.90	0.81	0.82
query16	0.36	0.37	0.38
query17	1.06	1.02	1.06
query18	0.19	0.19	0.19
query19	1.94	1.87	1.99
query20	0.01	0.01	0.01
query21	15.34	0.58	0.57
query22	2.83	2.89	1.79
query23	17.34	0.82	0.87
query24	2.96	0.78	1.08
query25	0.20	0.12	0.05
query26	0.42	0.13	0.14
query27	0.05	0.04	0.05
query28	10.84	1.10	1.07
query29	12.50	3.28	3.23
query30	0.24	0.06	0.06
query31	2.88	0.38	0.37
query32	3.28	0.46	0.46
query33	3.02	3.01	3.04
query34	16.83	4.41	4.35
query35	4.53	4.53	4.47
query36	0.66	0.51	0.50
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.5 s
Total hot run time: 32.56 s

github-actions · 2024-09-19T03:28:01Z

PR approved by anyone and no changes requested.

github-actions · 2024-09-19T03:38:10Z

PR approved by at least one committer and no changes requested.

… table (apache#40944) Fix unstable issue like: ``` 2024-09-17 12:41:04,755 WARN (thrift-server-pool-4|384) [FrontendServiceImpl.fetchSchemaTableData():2292] Failed to fetchSchemaTableData java.lang.NullPointerException: Cannot invoke "org.apache.doris.catalog.MaterializedIndexMeta.getSchema()" because the return value of "java.util.Map.get(Object)" is null at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:954) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:947) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getBaseSchema(OlapTable.java:2097) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getKeyColAsString(OlapTable.java:2167) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsForInternalCatalog(MetadataGenerator.java:1099) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsMetadataResult(MetadataGenerator.java:1189) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.getSchemaTableData(MetadataGenerator.java:273) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2289) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?] at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.proxy2.$Proxy44.fetchSchemaTableData(Unknown Source) ~[?:?] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4572) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4552) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ```

…table (#40944) (#41040) bp #40944

…rtitions sys table (#40972) ## Proposed changes Issue Number: close #xxx fix potential NPE when query table_properties and partitions sys table follow up for #40944

… table (apache#40944) Fix unstable issue like: ``` 2024-09-17 12:41:04,755 WARN (thrift-server-pool-4|384) [FrontendServiceImpl.fetchSchemaTableData():2292] Failed to fetchSchemaTableData java.lang.NullPointerException: Cannot invoke "org.apache.doris.catalog.MaterializedIndexMeta.getSchema()" because the return value of "java.util.Map.get(Object)" is null at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:954) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getSchemaByIndexId(OlapTable.java:947) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getBaseSchema(OlapTable.java:2097) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getKeyColAsString(OlapTable.java:2167) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsForInternalCatalog(MetadataGenerator.java:1099) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.tableOptionsMetadataResult(MetadataGenerator.java:1189) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.getSchemaTableData(MetadataGenerator.java:273) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2289) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?] at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.proxy2.$Proxy44.fetchSchemaTableData(Unknown Source) ~[?:?] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4572) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4552) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ```

…rtitions sys table (apache#40972) ## Proposed changes Issue Number: close #xxx fix potential NPE when query table_properties and partitions sys table follow up for apache#40944

) (#40972) (#40553) (#40198) (#40905) (#40757) (#41265) [fix](systable) fix unstable case for partitions table (#40553) [fix](sys_table) fix potential NPE when query table_properties and partitions sys table (#40972) [fix](table_options) fix potential NPE when quering table_options sys table (#40944) [fix](test-case) fix unstable test case about table_options sys table (#41006) [fix](external-p2) ignore external p2 cases (#41148) [fix](export) fix p2 tests of export (#40198) [Fix](test)Fixing Kerberos Principal Not Mapping Correctly to a Local User (#40905) [chore](test)Exclude Hive-related packages from `java-udf-src`. (#40757) --------- Co-authored-by: Vallish Pai <vallishpai@gmail.com> Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com> Co-authored-by: Calvin Kirs <kirs@apache.org>

[fix](table_options) fix potential NPE when quering table_options sys…

ca55c3b

… table

morningman marked this pull request as ready for review September 18, 2024 14:15

morningman added dev/2.1.x dev/3.0.x labels Sep 18, 2024

wuwenchi approved these changes Sep 19, 2024

View reviewed changes

github-actions bot added the reviewed label Sep 19, 2024

suxiaogang223 approved these changes Sep 19, 2024

View reviewed changes

CalvinKirs approved these changes Sep 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024

morningman merged commit 158f0b2 into apache:master Sep 19, 2024

Vallishp mentioned this pull request Sep 19, 2024

[fix](sys_table) fix potential NPE when query table_properties and partitions sys table #40972

Merged

morningman mentioned this pull request Sep 20, 2024

[fix](table_options) fix potential NPE when quering table_options systable (#40944) #41040

Merged

morningman added a commit that referenced this pull request Sep 20, 2024

[fix](table_options) fix potential NPE when quering table_options sys…

3484e7b

…table (#40944) (#41040) bp #40944

morningman added dev/2.1.7-merged and removed dev/2.1.x labels Sep 20, 2024

gavinchou added dev/3.0.2-merged and removed dev/3.0.x labels Sep 25, 2024

morningman mentioned this pull request Sep 25, 2024

[pick] pick some pr to fix unstable test cases (#41148) (#41006) (#40944) (#40972) (#40553) (#40198) (#40905) (#40757) #41265

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](table_options) fix potential NPE when quering table_options sys table#40944

[fix](table_options) fix potential NPE when quering table_options sys table#40944
morningman merged 1 commit intoapache:masterfrom
morningman:fix_p0_table_opt

morningman commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

morningman commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

morningman commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

morningman commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

doris-robot commented Sep 18, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants