Skip to content

[Fix](Variant Type) forbit distribution info contains variant columns#33707

Merged
xiaokang merged 1 commit intoapache:masterfrom
eldenmoon:var-f
Apr 17, 2024
Merged

[Fix](Variant Type) forbit distribution info contains variant columns#33707
xiaokang merged 1 commit intoapache:masterfrom
eldenmoon:var-f

Conversation

@eldenmoon
Copy link
Copy Markdown
Member

@eldenmoon eldenmoon commented Apr 16, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Copy Markdown
Member Author

eldenmoon commented Apr 16, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 16, 2024

TPC-H: Total hot run time: 38311 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b0a6a885e64deeb4a317e4190999eac9e7832e2, data reload: false

------ Round 1 ----------------------------------
q1	17609	4260	4185	4185
q2	2015	201	184	184
q3	10454	1141	1170	1141
q4	10202	858	738	738
q5	7539	2643	2630	2630
q6	216	133	134	133
q7	1007	607	575	575
q8	9240	2045	2008	2008
q9	7450	6619	6498	6498
q10	8583	3527	3499	3499
q11	453	230	231	230
q12	408	219	224	219
q13	17768	2939	2990	2939
q14	269	224	224	224
q15	511	481	482	481
q16	523	378	386	378
q17	954	732	731	731
q18	7287	6867	6656	6656
q19	7736	1538	1508	1508
q20	642	321	305	305
q21	3538	2775	2745	2745
q22	355	305	304	304
Total cold run time: 114759 ms
Total hot run time: 38311 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4198	4213	4198
q2	373	264	269	264
q3	2978	2792	2747	2747
q4	1843	1581	1563	1563
q5	5275	5272	5291	5272
q6	206	123	123	123
q7	2227	1863	1870	1863
q8	3184	3334	3318	3318
q9	8579	8600	8653	8600
q10	4054	3858	4015	3858
q11	603	498	500	498
q12	781	650	628	628
q13	16444	3191	3163	3163
q14	308	286	286	286
q15	536	486	485	485
q16	491	430	453	430
q17	1799	1576	1520	1520
q18	8050	8020	7852	7852
q19	1690	1543	1562	1543
q20	2001	1852	1833	1833
q21	9873	5035	4934	4934
q22	563	452	466	452
Total cold run time: 76216 ms
Total hot run time: 55430 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 16, 2024

TPC-DS: Total hot run time: 185481 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b0a6a885e64deeb4a317e4190999eac9e7832e2, data reload: false

query1	893	372	360	360
query2	6627	2570	2337	2337
query3	6666	207	204	204
query4	23195	21272	21472	21272
query5	4229	418	435	418
query6	279	190	206	190
query7	4584	291	294	291
query8	246	177	180	177
query9	8660	2363	2340	2340
query10	414	247	248	247
query11	14793	14192	14192	14192
query12	138	96	88	88
query13	1657	369	355	355
query14	9883	8024	7962	7962
query15	253	204	183	183
query16	8061	261	267	261
query17	1959	588	561	561
query18	2083	286	283	283
query19	329	148	155	148
query20	93	82	83	82
query21	196	133	129	129
query22	5121	4894	4880	4880
query23	33872	33308	33302	33302
query24	7220	3058	3107	3058
query25	599	387	404	387
query26	710	156	157	156
query27	2279	367	367	367
query28	5678	2086	2095	2086
query29	885	628	624	624
query30	334	184	181	181
query31	971	745	779	745
query32	93	52	53	52
query33	572	256	256	256
query34	897	492	493	492
query35	813	719	718	718
query36	1059	916	918	916
query37	121	84	78	78
query38	3420	3354	3331	3331
query39	1643	1615	1600	1600
query40	177	138	135	135
query41	49	46	52	46
query42	111	102	98	98
query43	597	552	540	540
query44	1110	768	758	758
query45	297	284	280	280
query46	1111	760	748	748
query47	2032	1908	1942	1908
query48	399	319	319	319
query49	828	401	392	392
query50	809	424	416	416
query51	6924	6871	6741	6741
query52	109	85	88	85
query53	345	284	281	281
query54	257	223	229	223
query55	76	74	70	70
query56	252	241	257	241
query57	1192	1128	1130	1128
query58	215	200	202	200
query59	3407	3034	3060	3034
query60	257	240	237	237
query61	91	88	88	88
query62	588	435	436	435
query63	308	284	280	280
query64	4828	4024	3867	3867
query65	3058	3030	3023	3023
query66	736	344	335	335
query67	15421	15083	15086	15083
query68	6378	538	535	535
query69	540	310	306	306
query70	1249	1187	1128	1128
query71	445	263	275	263
query72	6419	2678	2469	2469
query73	733	324	323	323
query74	6902	6461	6377	6377
query75	3591	2591	2662	2591
query76	3718	983	938	938
query77	629	269	277	269
query78	10841	10288	10249	10249
query79	7919	511	528	511
query80	1491	431	433	431
query81	523	247	245	245
query82	1558	94	100	94
query83	202	168	165	165
query84	265	86	80	80
query85	1266	273	261	261
query86	446	300	288	288
query87	3469	3291	3281	3281
query88	5149	2415	2448	2415
query89	499	374	394	374
query90	1857	183	188	183
query91	127	97	99	97
query92	59	48	48	48
query93	6660	500	493	493
query94	1096	183	182	182
query95	406	296	300	296
query96	596	261	260	260
query97	3140	2956	2959	2956
query98	238	226	217	217
query99	1223	860	856	856
Total cold run time: 286743 ms
Total hot run time: 185481 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 16, 2024

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b0a6a885e64deeb4a317e4190999eac9e7832e2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.06	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.49
query6	1.46	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.50
query10	0.54	0.56	0.55
query11	0.16	0.12	0.12
query12	0.14	0.12	0.11
query13	0.60	0.58	0.58
query14	0.76	0.78	0.78
query15	0.83	0.81	0.81
query16	0.37	0.36	0.37
query17	1.00	0.99	1.02
query18	0.23	0.24	0.23
query19	1.75	1.66	1.72
query20	0.02	0.01	0.01
query21	15.40	0.67	0.66
query22	3.94	7.20	1.99
query23	18.33	1.34	1.31
query24	2.06	0.22	0.21
query25	0.13	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.27	1.01	0.98
query29	12.58	3.29	3.23
query30	0.25	0.06	0.07
query31	2.86	0.38	0.38
query32	3.26	0.48	0.47
query33	2.85	2.82	2.87
query34	17.03	4.41	4.43
query35	4.47	4.48	4.48
query36	0.65	0.46	0.46
query37	0.18	0.16	0.17
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.17	0.13	0.16
query41	0.09	0.05	0.05
query42	0.05	0.05	0.06
query43	0.04	0.03	0.04
Total cold run time: 109.12 s
Total hot run time: 30.53 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 16, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3b0a6a885e64deeb4a317e4190999eac9e7832e2 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      34 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 16, 2024

PR approved by anyone and no changes requested.

xiaokang
xiaokang approved these changes Apr 17, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 17, 2024

PR approved by at least one committer and no changes requested.

@xiaokang xiaokang merged commit 21230a3 into apache:master Apr 17, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Apr 18, 2024
@eldenmoon
[Fix](Variant Type) forbit distribution info contains variant columns (…
fd03227 
…apache#33707)
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon @Doris-Extras
[Fix](Variant Type) forbit distribution info contains variant columns (…
4de357c 
…#33707)
@eldenmoon eldenmoon mentioned this pull request Apr 18, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon
[Pick](Variant) pick #33734 #33766 #33707 to branch-2.1 (#33848)
8f6f4cf 
* [Fix](Variant Type) forbit distribution info contains variant columns (#33707)

* [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size (#33734)

insert_range_from should start from `size` with `count` elements for null map

* [Fix](Variant) check column index validation for extracted columns (#33766)
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@eldenmoon @dataroaring
[Fix](Variant Type) forbit distribution info contains variant columns (…
a40dc9a 
…#33707)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@eldenmoon @doris-robot @xiaokang @zclllyybb