Skip to content

[Fix](Variant) ensure variant column finalized before reading the root column#37674

Merged
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:fix-var-partial-update
Jul 16, 2024
Merged

[Fix](Variant) ensure variant column finalized before reading the root column#37674
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:fix-var-partial-update

Conversation

@eldenmoon
Copy link
Copy Markdown
Member

@eldenmoon eldenmoon commented Jul 11, 2024

…t column

Eg. segment 1 has no variant columns and fill the dst with default value, which the variant columns's root is type nothing with n rows(not finalized), then reading segment2 with the none finalized variant column will lead to nullptr in obj.get_root();

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Copy Markdown
Member Author

eldenmoon commented Jul 11, 2024

run buildall

@eldenmoon
[Fix](Variant) ensure variant column finalized before reading the roo…
9294447 
…t column

Eg. segment 1 has no variant columns and fill the dst with default value, which the variant columns's root is type nothing with n rows(not finalized), then
reading segment2 with the none finalized variant column will lead to nullptr in `obj.get_root();`
@eldenmoon eldenmoon force-pushed the fix-var-partial-update branch from a42d034 to 9294447 Compare July 11, 2024 10:32
@eldenmoon
Copy link
Copy Markdown
Member Author

eldenmoon commented Jul 11, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

TPC-H: Total hot run time: 40091 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92944471dcdafe2422a78df661bd322914b0e30e, data reload: false

------ Round 1 ----------------------------------
q1	18113	4450	4421	4421
q2	2022	185	181	181
q3	10475	1169	1133	1133
q4	10198	754	779	754
q5	7518	2674	2774	2674
q6	222	136	137	136
q7	974	595	592	592
q8	9213	2059	2069	2059
q9	8626	6546	6603	6546
q10	8632	3728	3736	3728
q11	455	232	235	232
q12	403	227	215	215
q13	17761	2958	2959	2958
q14	293	238	240	238
q15	541	483	471	471
q16	464	384	373	373
q17	959	739	790	739
q18	8094	7702	7376	7376
q19	6483	1451	1513	1451
q20	664	321	323	321
q21	4832	3167	3170	3167
q22	387	326	338	326
Total cold run time: 117329 ms
Total hot run time: 40091 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4408	4284	4236	4236
q2	375	281	267	267
q3	3034	2732	2748	2732
q4	1837	1580	1569	1569
q5	5278	5302	5312	5302
q6	222	130	130	130
q7	2079	1758	1739	1739
q8	3201	3341	3319	3319
q9	8358	8361	8331	8331
q10	3868	3710	3676	3676
q11	571	498	488	488
q12	799	617	620	617
q13	16638	2976	2966	2966
q14	306	269	271	269
q15	521	479	489	479
q16	468	411	424	411
q17	1764	1486	1474	1474
q18	7571	7451	7330	7330
q19	1686	1619	1593	1593
q20	2012	1804	1799	1799
q21	4720	4627	4645	4627
q22	612	547	557	547
Total cold run time: 70328 ms
Total hot run time: 53901 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

TPC-DS: Total hot run time: 173919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92944471dcdafe2422a78df661bd322914b0e30e, data reload: false

query1	901	371	367	367
query2	6459	2434	2311	2311
query3	6687	211	222	211
query4	27650	17500	17495	17495
query5	4235	474	469	469
query6	260	171	166	166
query7	4599	285	289	285
query8	322	307	300	300
query9	8631	2443	2438	2438
query10	446	281	280	280
query11	11678	9897	10076	9897
query12	144	82	82	82
query13	1661	383	374	374
query14	9677	7715	7634	7634
query15	265	189	199	189
query16	7695	343	321	321
query17	1776	553	540	540
query18	1955	289	283	283
query19	206	173	153	153
query20	89	84	85	84
query21	208	127	122	122
query22	4418	4106	4034	4034
query23	33776	32950	33098	32950
query24	11971	2934	2823	2823
query25	663	387	392	387
query26	1664	145	151	145
query27	2896	277	279	277
query28	7768	2086	2080	2080
query29	1010	660	622	622
query30	288	155	149	149
query31	948	760	750	750
query32	99	56	61	56
query33	779	318	313	313
query34	974	516	498	498
query35	696	581	573	573
query36	1096	969	921	921
query37	248	80	81	80
query38	2880	2762	2736	2736
query39	848	793	813	793
query40	287	123	122	122
query41	55	52	54	52
query42	122	97	106	97
query43	594	546	545	545
query44	1190	714	718	714
query45	205	173	170	170
query46	1078	743	736	736
query47	1870	1817	1799	1799
query48	374	303	304	303
query49	1208	418	507	418
query50	770	399	389	389
query51	6849	6776	6785	6776
query52	107	91	90	90
query53	357	284	287	284
query54	851	452	465	452
query55	74	74	72	72
query56	285	270	264	264
query57	1178	1031	1042	1031
query58	252	234	239	234
query59	3263	3060	3122	3060
query60	312	282	282	282
query61	108	95	97	95
query62	818	652	665	652
query63	324	299	288	288
query64	10569	2167	1692	1692
query65	3236	3117	3134	3117
query66	1411	331	324	324
query67	15574	15115	15043	15043
query68	5589	543	534	534
query69	703	442	351	351
query70	1134	1134	1049	1049
query71	472	280	282	280
query72	8773	5352	5657	5352
query73	780	329	329	329
query74	6031	5542	5490	5490
query75	4342	2693	2661	2661
query76	3949	952	916	916
query77	678	311	302	302
query78	9666	9051	9613	9051
query79	7456	525	525	525
query80	970	474	469	469
query81	579	220	221	220
query82	1597	134	132	132
query83	266	165	167	165
query84	277	85	87	85
query85	1554	299	338	299
query86	450	332	322	322
query87	3290	3096	3158	3096
query88	5264	2498	2489	2489
query89	514	385	388	385
query90	1860	197	202	197
query91	137	103	105	103
query92	63	51	51	51
query93	5374	506	497	497
query94	1204	220	216	216
query95	400	323	330	323
query96	609	279	274	274
query97	3121	3009	3004	3004
query98	224	200	192	192
query99	1549	1272	1271	1271
Total cold run time: 300762 ms
Total hot run time: 173919 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 11, 2024

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92944471dcdafe2422a78df661bd322914b0e30e, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.68	0.09	0.08
query5	0.50	0.48	0.49
query6	1.15	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.54	0.56
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.58
query14	0.76	0.79	0.80
query15	0.84	0.80	0.81
query16	0.37	0.39	0.37
query17	1.01	1.00	1.06
query18	0.22	0.22	0.21
query19	1.84	1.83	1.70
query20	0.01	0.01	0.01
query21	15.43	0.74	0.66
query22	4.46	6.84	1.94
query23	18.34	1.41	1.27
query24	2.08	0.22	0.24
query25	0.17	0.09	0.09
query26	0.29	0.21	0.21
query27	0.46	0.23	0.23
query28	13.27	1.02	0.99
query29	12.64	3.33	3.28
query30	0.25	0.06	0.05
query31	2.86	0.41	0.38
query32	3.30	0.47	0.46
query33	2.87	2.90	2.93
query34	17.15	4.31	4.31
query35	4.43	4.40	4.41
query36	0.64	0.47	0.51
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.17 s
Total hot run time: 30.64 s

@xiaokang xiaokang changed the title [Fix](Variant) ensure variant column finalized before reading the roo… [Fix](Variant) ensure variant column finalized before reading the root column Jul 11, 2024
@xiaokang xiaokang added the p0_c label Jul 15, 2024
xiaokang
xiaokang approved these changes Jul 16, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 16, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 16, 2024

PR approved by anyone and no changes requested.

lxr599
lxr599 approved these changes Jul 16, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@lxr599 lxr599 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 257b35d into apache:master Jul 16, 2024
@eldenmoon eldenmoon removed the doing label Jul 16, 2024
@xiaokang xiaokang added the doing label Jul 16, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 16, 2024
@eldenmoon
[Fix](Variant) ensure variant column finalized before reading the roo…
4f62297 
…t column (apache#37674)

…t column

Eg. segment 1 has no variant columns and fill the dst with default
value, which the variant columns's root is type nothing with n rows(not
finalized), then reading segment2 with the none finalized variant column
will lead to nullptr in `obj.get_root();`
@eldenmoon eldenmoon mentioned this pull request Jul 16, 2024
Merged
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 16, 2024
@eldenmoon
[Fix](Variant) ensure variant column finalized before reading the roo…
c2b4ef1 
…t column (apache#37674)

…t column

Eg. segment 1 has no variant columns and fill the dst with default
value, which the variant columns's root is type nothing with n rows(not
finalized), then reading segment2 with the none finalized variant column
will lead to nullptr in `obj.get_root();`
@eldenmoon eldenmoon mentioned this pull request Jul 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@eldenmoon
[Pick](Variant) pick some fix (#37922)
b15ccdb 
#37674
#37839
#37883 
#37857 
#37794
xiaokang pushed a commit that referenced this pull request Jul 17, 2024
@eldenmoon
[Pick](Variant) pick some fix (#37926)
2994232 
#37674
#37839
#37883
#37857
#37794
#37640
#37557
#37217
#37121
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@eldenmoon @seawinde
[Fix](Variant) ensure variant column finalized before reading the roo…
39a416f 
…t column (apache#37674)

…t column

Eg. segment 1 has no variant columns and fill the dst with default
value, which the variant columns's root is type nothing with n rows(not
finalized), then reading segment2 with the none finalized variant column
will lead to nullptr in `obj.get_root();`
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

+1 more reviewer

@lxr599 lxr599 lxr599 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged p0_c reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@eldenmoon @doris-robot @xiaokang @lxr599 @yiguolei