Skip to content

[opt](autoinc) Forbid some schema change when the table has auto-increment column#37186

Merged
dataroaring merged 1 commit intoapache:masterfrom
bobhan1:forbid-autoinc-schema-change
Jul 4, 2024
Merged

[opt](autoinc) Forbid some schema change when the table has auto-increment column#37186
dataroaring merged 1 commit intoapache:masterfrom
bobhan1:forbid-autoinc-schema-change

Conversation

@bobhan1
Copy link
Copy Markdown
Contributor

@bobhan1 bobhan1 commented Jul 2, 2024 •
edited
Loading

Proposed changes

  1. forbid schema change that modify the auto-increment attribute of a column.
  2. forbid adding auto-increment column to a table.
  3. forbid schema change that invoke double writes on table which has auto-increment column.

branch-2.1-pick: #37331

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 2, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Copy Markdown
Contributor Author

bobhan1 commented Jul 2, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 2, 2024

TPC-H: Total hot run time: 39801 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83fdfd0604753725db082d624f8fbd624036fb31, data reload: false

------ Round 1 ----------------------------------
q1	17904	4495	4353	4353
q2	3043	195	200	195
q3	11738	1172	1130	1130
q4	10614	844	829	829
q5	7650	2692	2644	2644
q6	229	142	137	137
q7	979	601	614	601
q8	9447	2072	2066	2066
q9	8785	6483	6425	6425
q10	9005	3734	3701	3701
q11	462	232	232	232
q12	423	224	220	220
q13	17900	2945	2984	2945
q14	251	227	221	221
q15	509	485	492	485
q16	505	371	372	371
q17	972	637	682	637
q18	8086	7461	7368	7368
q19	6029	1558	1483	1483
q20	650	330	317	317
q21	4901	3107	3764	3107
q22	394	338	334	334
Total cold run time: 120476 ms
Total hot run time: 39801 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4414	4335	4251	4251
q2	367	265	252	252
q3	2965	2731	2737	2731
q4	1904	1633	1599	1599
q5	5229	5243	5235	5235
q6	220	130	130	130
q7	2113	1722	1718	1718
q8	3176	3363	3353	3353
q9	8292	8332	8331	8331
q10	3868	3732	3707	3707
q11	592	495	486	486
q12	797	599	596	596
q13	17365	2979	2976	2976
q14	280	259	267	259
q15	527	479	483	479
q16	463	417	417	417
q17	1771	1473	1469	1469
q18	7632	7444	7315	7315
q19	1722	1559	1618	1559
q20	1993	1797	1793	1793
q21	4779	4687	4632	4632
q22	637	564	531	531
Total cold run time: 71106 ms
Total hot run time: 53819 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 2, 2024

TPC-DS: Total hot run time: 172515 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83fdfd0604753725db082d624f8fbd624036fb31, data reload: false

query1	918	374	360	360
query2	6458	2445	2307	2307
query3	6661	201	211	201
query4	19768	17456	17241	17241
query5	4204	481	468	468
query6	264	191	159	159
query7	4593	288	292	288
query8	335	302	294	294
query9	8524	2371	2350	2350
query10	608	321	295	295
query11	10369	9903	9994	9903
query12	137	85	83	83
query13	1647	381	373	373
query14	9538	7806	6979	6979
query15	237	189	186	186
query16	7832	280	277	277
query17	1877	551	533	533
query18	1915	283	282	282
query19	201	154	161	154
query20	92	88	86	86
query21	218	137	133	133
query22	4221	4170	3960	3960
query23	33415	32825	33268	32825
query24	11024	2808	2820	2808
query25	669	387	400	387
query26	1428	159	152	152
query27	2990	321	321	321
query28	7200	2070	2047	2047
query29	963	640	624	624
query30	289	158	155	155
query31	995	754	755	754
query32	88	53	56	53
query33	781	318	303	303
query34	902	480	478	478
query35	753	662	604	604
query36	1111	941	960	941
query37	154	78	77	77
query38	2850	2754	2765	2754
query39	862	811	776	776
query40	213	128	127	127
query41	59	55	52	52
query42	119	97	108	97
query43	588	574	526	526
query44	1261	737	745	737
query45	191	163	167	163
query46	1074	701	734	701
query47	1844	1769	1773	1769
query48	363	308	306	306
query49	1119	423	433	423
query50	761	392	395	392
query51	6852	6854	6678	6678
query52	105	98	97	97
query53	360	301	299	299
query54	881	458	472	458
query55	78	74	76	74
query56	318	286	286	286
query57	1160	1060	1059	1059
query58	296	254	259	254
query59	3456	3291	3167	3167
query60	310	287	284	284
query61	89	93	96	93
query62	637	427	437	427
query63	321	292	294	292
query64	8967	2278	1739	1739
query65	3136	3100	3060	3060
query66	961	333	322	322
query67	15616	14789	14951	14789
query68	6486	560	553	553
query69	733	455	351	351
query70	1219	1128	1145	1128
query71	531	279	287	279
query72	8817	5487	5526	5487
query73	817	329	326	326
query74	5971	5413	5460	5413
query75	5039	2686	2702	2686
query76	4806	975	966	966
query77	806	294	304	294
query78	10278	9793	9762	9762
query79	8230	509	515	509
query80	1045	479	478	478
query81	551	226	224	224
query82	839	104	101	101
query83	368	168	164	164
query84	270	86	85	85
query85	1413	271	299	271
query86	443	339	320	320
query87	3251	3051	3089	3051
query88	4903	2421	2418	2418
query89	564	395	383	383
query90	2025	182	188	182
query91	129	98	98	98
query92	63	47	49	47
query93	6885	525	511	511
query94	1290	186	185	185
query95	400	315	317	315
query96	603	275	276	275
query97	3126	2980	2971	2971
query98	222	202	196	196
query99	1151	847	844	844
Total cold run time: 291500 ms
Total hot run time: 172515 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 2, 2024

ClickBench: Total hot run time: 30.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83fdfd0604753725db082d624f8fbd624036fb31, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.49	0.48
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.56	0.49	0.48
query10	0.53	0.53	0.53
query11	0.16	0.12	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.59
query14	0.76	0.79	0.77
query15	0.84	0.81	0.82
query16	0.36	0.36	0.36
query17	0.99	0.98	1.02
query18	0.24	0.23	0.26
query19	1.78	1.69	1.78
query20	0.01	0.00	0.01
query21	15.45	0.76	0.65
query22	4.64	7.39	1.93
query23	18.29	1.34	1.32
query24	2.10	0.24	0.22
query25	0.15	0.09	0.09
query26	0.27	0.17	0.17
query27	0.08	0.09	0.08
query28	13.24	1.02	0.99
query29	12.64	3.27	3.25
query30	0.25	0.06	0.06
query31	2.89	0.39	0.39
query32	3.27	0.46	0.46
query33	2.93	2.84	2.89
query34	17.11	4.37	4.46
query35	4.44	4.44	4.45
query36	0.64	0.46	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.83 s
Total hot run time: 30.44 s

dataroaring
dataroaring approved these changes Jul 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 3, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 3, 2024

PR approved by anyone and no changes requested.

@Lchangliang
Copy link
Copy Markdown
Contributor

Lchangliang commented Jul 3, 2024

Why need to do that?

@dataroaring dataroaring merged commit d790599 into apache:master Jul 4, 2024
@bobhan1 bobhan1 mentioned this pull request Jul 5, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 5, 2024
@bobhan1
[branch-2.1] Picks "[opt](autoinc) Forbid some schema change when the…
5de6aa7 
… table has auto-increment column #37186" (#37331)

## Proposed changes

picks #37186
bobhan1 added a commit to bobhan1/doris that referenced this pull request Jul 9, 2024
@bobhan1
Forbid some schema change when the table has auto-increment column ap…
45a9e36 
…ache#37186
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@bobhan1 @dataroaring
[opt](autoinc) Forbid some schema change when the table has auto-incr…
8c7104b 
…ement column (#37186)

## Proposed changes
1. forbid schema change that modify the auto-increment attribute of a
column.
2. forbid adding auto-increment column to a table.
3. forbid schema change that invoke double writes on table which has
auto-increment column.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@bobhan1 bobhan1 mentioned this pull request Sep 3, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 24, 2024
@bobhan1
[opt](autoinc) Remove some restrictions on schema change on table tha…
1593b4d 
…t has auto-increment column (#40280)

## Proposed changes

- #37186 forbids some schema change on table with auto-increment column,
one of which is to forbid schema change that involves double write on
tables which has auto-increment column. However, the process of
generating auto-increment column values is before the distributing data
to indexes. So this restriction can be removed.

branch-2.1-pick: #41096
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@bobhan1 @dataroaring
[opt](autoinc) Remove some restrictions on schema change on table tha…
15a26bb 
…t has auto-increment column (#40280)

## Proposed changes

- #37186 forbids some schema change on table with auto-increment column,
one of which is to forbid schema change that involves double write on
tables which has auto-increment column. However, the process of
generating auto-increment column values is before the distributing data
to indexes. So this restriction can be removed.

branch-2.1-pick: #41096
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bobhan1 @doris-robot @Lchangliang @Yukang-Lian @dataroaring