[feature](txn insert) txn insert support insert into select#31666
Merged
dataroaring merged 2 commits intoapache:masterfrom Mar 10, 2024
Merged
[feature](txn insert) txn insert support insert into select#31666dataroaring merged 2 commits intoapache:masterfrom
dataroaring merged 2 commits intoapache:masterfrom
Conversation
|
Thank you for your contribution to Apache Doris. |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
3becfa8 to
e40a1ab
Compare
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32116 ms |
TPC-DS: Total hot run time: 187669 ms |
ClickBench: Total hot run time: 27.03 s |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
f18dccc to
659ca52
Compare
Contributor
Author
|
run buildall |
morrySnow
reviewed
Mar 4, 2024
| } else { | ||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel()) { | ||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel() | ||
| && sink.child() instanceof LogicalInlineTable) { |
Contributor
There was a problem hiding this comment.
In legacy planner, we support insert into t select 1. So if we only support LogicalInlineTable here means we have diff with legacy planner
Contributor
Author
There was a problem hiding this comment.
The nereids does not support insert into t select 1? It fallback to legacy
| } | ||
| OlapTable olapTable = (OlapTable) targetTableIf; | ||
| String label = this.labelName.orElse( | ||
| ctx.isTxnModel() ? null : String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo)); |
Contributor
There was a problem hiding this comment.
could u add some comment to explain why lable should be null when we in txn model
| TransactionEntry txnEntry = ctx.getTxnEntry(); | ||
| if (txnEntry.isTransactionBegan()) { | ||
| throw new AnalysisException( | ||
| "Transaction insert can not insert into values and insert into select at the same time"); |
Contributor
There was a problem hiding this comment.
why isTransactionBegan means use insert into values with insert into select at same time?
Contributor
Author
There was a problem hiding this comment.
isTransactionBegan means the insert into select already began a txn.
This function will begin a txn for insert into values.
Currently we forbid it, maybe support it later.
Contributor
There was a problem hiding this comment.
add a TODO coment here, we'll support mix usage of insert into values and insert into select in future.
| boolean isGroupCommit = (parsedStmt != null | ||
| && parsedStmt instanceof LogicalPlanAdapter | ||
| && ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand) | ||
| && !context.isTxnModel(); |
Contributor
There was a problem hiding this comment.
so all insert into without txn model is group commit?
Contributor
Author
There was a problem hiding this comment.
The logic should be
boolean isGroupCommit = (Config.wait_internal_group_commit_finish
|| context.sessionVariable.isEnableInsertGroupCommit()) && parsedStmt != null
&& parsedStmt instanceof LogicalPlanAdapter
&& ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand;
but some nereids case can not pass.
I add !context.isTxnModel() here becase the regression cases expect txn insert does not force fallback
Contributor
There was a problem hiding this comment.
so the variable name now should not be isGroupCommit? please use a better variable name, and add comments to explain it here
zhannngchen
reviewed
Mar 5, 2024
zhannngchen
reviewed
Mar 5, 2024
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
659ca52 to
96f04b2
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 37666 ms |
TPC-DS: Total hot run time: 178038 ms |
ClickBench: Total hot run time: 30.64 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
96f04b2 to
2bf8bd0
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
run p0 |
morningman
pushed a commit
that referenced
this pull request
Apr 12, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 5, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Transaction insert support:
Limit:
insert into selectandinsert into valuescan be used at one transaction.Will support in later pr.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...