Skip to content

fix restore with partitions cause some query fail#8245

Merged
morningman merged 2 commits intoapache:masterfrom
GoGoWen:restore_enhance
Mar 3, 2022
Merged

fix restore with partitions cause some query fail#8245
morningman merged 2 commits intoapache:masterfrom
GoGoWen:restore_enhance

Conversation

@GoGoWen
Copy link
Copy Markdown
Contributor

@GoGoWen GoGoWen commented Feb 25, 2022

Proposed changes

Issue Number: close #8244

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

morningman
morningman reviewed Feb 26, 2022
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
if (partition == null) {
continue;
}
partition.setState(PartitionState.RESTORE);
Copy link
Copy Markdown
Contributor

@morningman morningman Feb 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you also need to modify replayCheckAndPrepareMeta() method.

@GoGoWen GoGoWen requested a review from morningman February 28, 2022 10:00
@jackwener jackwener added the kind/fix Categorizes issue or PR as related to a bug. label Feb 28, 2022
morningman
morningman approved these changes Mar 1, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added the dev/backlog waiting to be merged in future dev branch label Mar 1, 2022
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 1, 2022
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 1, 2022

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 1, 2022

PR approved by anyone and no changes requested.

@morningman morningman merged commit 80e8815 into apache:master Mar 3, 2022
morningman pushed a commit that referenced this pull request Mar 14, 2022
@GoGoWen @morningman-cmy
[improvement](restore) allow query on part of partitions when others …
ae45eed 
…are in RESTORE (#8245)
@morningman morningman added dev/merged-1.0.1-deprecated PR has been merged into dev-1.0.1 and removed dev/backlog waiting to be merged in future dev branch labels Mar 14, 2022
This was referenced Aug 20, 2024
Merged
Merged
w41ter pushed a commit that referenced this pull request Aug 26, 2024
@Johnnyssc
[enhancement](Load)allow load data to the other partitions when some …
6580981 
…partitions are restoring (#39595)

If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

ps: set restore status for partitions in this
pr:#8245


## test case for this pr

### restore tbl's partition p202408
$ RESTORE SNAPSHOT db.tbl_p202408_test
FROM repo
ON(
    `tbl` PARTITION (p202408)
)
PROPERTIES(
    "backup_timestamp"="2024-08-22-20-32-37",
    "replication_num" = "1"
);

###  check restore job state\G
$ SHOW RESTORE\G
*************************** 1. row ***************************
JobId: 21741
Label: tbl_p202408_test
Timestamp: 2024-08-22-20-32-37
State: DOWNLOADING
RestoreObjs: {
  "name": "tbl_p202408_test",
  "database": "db",
  "olap_table_list": [
      {
          "name": "tbl",
          "partition_names": ["p202408"]
      }
]

### load to partition p202408, failed with exception
 curl --location-trusted -u root:"" \
>   -H "label:tbl_test_load_19" \
>   -H "timeout:300" \
>   -H "format: parquet" \
>   -T data_for_p202408.parquet \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3042,
    "Label": "tbl_test_load_19",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Table
[zt_order_detail_v3], Partition [p202408] is in restore process. Can not
load into it.etc.",
    "NumberTotalRows": 682,
    "NumberLoadedRows": 682,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 82025,
    "LoadTimeMs": 48,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 38,
    "CommitAndPublishTimeMs": 0
}

### load to partition p202408, successfully
$ curl --location-trusted -u root:"" \
>   -H "timeout:300" \
>   -H "format: json" \
>   -H "read_json_by_line:true" \
>   -T  data_for_p202407.json \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3043,
    "Label": "2f2dae38-a495-4c22-9492-419ea70b724e",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1128,
    "LoadTimeMs": 51,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 6,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 30,
    "CommitAndPublishTimeMs": 13
}

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
w41ter pushed a commit that referenced this pull request Aug 26, 2024
@Johnnyssc
[enhancement](Load)allow load data to the other partitions when some …
ab4d81a 
…partitions are restoring (#39411)

If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

ps: set restore status for partitions in this
pr:#8245


## test case for this pr

### restore tbl's partition p202408
$ RESTORE SNAPSHOT db.tbl_p202408_test
FROM repo
ON(
    `tbl` PARTITION (p202408)
)
PROPERTIES(
    "backup_timestamp"="2024-08-22-20-32-37",
    "replication_num" = "1"
);

###  check restore job state\G
$ SHOW RESTORE\G
*************************** 1. row ***************************
JobId: 21741
Label: tbl_p202408_test
Timestamp: 2024-08-22-20-32-37
State: DOWNLOADING
RestoreObjs: {
  "name": "tbl_p202408_test",
  "database": "db",
  "olap_table_list": [
      {
          "name": "tbl",
          "partition_names": ["p202408"]
      }
]

### load to partition p202408, failed with exception
 curl --location-trusted -u root:"" \
>   -H "label:tbl_test_load_19" \
>   -H "timeout:300" \
>   -H "format: parquet" \
>   -T data_for_p202408.parquet \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3042,
    "Label": "tbl_test_load_19",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Table
[zt_order_detail_v3], Partition [p202408] is in restore process. Can not
load into it.etc.",
    "NumberTotalRows": 682,
    "NumberLoadedRows": 682,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 82025,
    "LoadTimeMs": 48,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 38,
    "CommitAndPublishTimeMs": 0
}

### load to partition p202408, successfully
$ curl --location-trusted -u root:"" \
>   -H "timeout:300" \
>   -H "format: json" \
>   -H "read_json_by_line:true" \
>   -T  data_for_p202407.json \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3043,
    "Label": "2f2dae38-a495-4c22-9492-419ea70b724e",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1128,
    "LoadTimeMs": 51,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 6,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 30,
    "CommitAndPublishTimeMs": 13
}

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@Johnnyssc @dataroaring
[enhancement](Load)allow load data to the other partitions when some …
1c43ee7 
…partitions are restoring (#39595)

If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

ps: set restore status for partitions in this
pr:#8245


## test case for this pr

### restore tbl's partition p202408
$ RESTORE SNAPSHOT db.tbl_p202408_test
FROM repo
ON(
    `tbl` PARTITION (p202408)
)
PROPERTIES(
    "backup_timestamp"="2024-08-22-20-32-37",
    "replication_num" = "1"
);

###  check restore job state\G
$ SHOW RESTORE\G
*************************** 1. row ***************************
JobId: 21741
Label: tbl_p202408_test
Timestamp: 2024-08-22-20-32-37
State: DOWNLOADING
RestoreObjs: {
  "name": "tbl_p202408_test",
  "database": "db",
  "olap_table_list": [
      {
          "name": "tbl",
          "partition_names": ["p202408"]
      }
]

### load to partition p202408, failed with exception
 curl --location-trusted -u root:"" \
>   -H "label:tbl_test_load_19" \
>   -H "timeout:300" \
>   -H "format: parquet" \
>   -T data_for_p202408.parquet \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3042,
    "Label": "tbl_test_load_19",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Table
[zt_order_detail_v3], Partition [p202408] is in restore process. Can not
load into it.etc.",
    "NumberTotalRows": 682,
    "NumberLoadedRows": 682,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 82025,
    "LoadTimeMs": 48,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 38,
    "CommitAndPublishTimeMs": 0
}

### load to partition p202408, successfully
$ curl --location-trusted -u root:"" \
>   -H "timeout:300" \
>   -H "format: json" \
>   -H "read_json_by_line:true" \
>   -T  data_for_p202407.json \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3043,
    "Label": "2f2dae38-a495-4c22-9492-419ea70b724e",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1128,
    "LoadTimeMs": 51,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 6,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 30,
    "CommitAndPublishTimeMs": 13
}

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
@Johnnyssc Johnnyssc mentioned this pull request Sep 12, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@Johnnyssc
[enhancement](Load)allow load data to the other partitions when some …
917306c 
…partitions are restoring (apache#39411)

If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

ps: set restore status for partitions in this
pr:apache#8245


## test case for this pr

### restore tbl's partition p202408
$ RESTORE SNAPSHOT db.tbl_p202408_test
FROM repo
ON(
    `tbl` PARTITION (p202408)
)
PROPERTIES(
    "backup_timestamp"="2024-08-22-20-32-37",
    "replication_num" = "1"
);

###  check restore job state\G
$ SHOW RESTORE\G
*************************** 1. row ***************************
JobId: 21741
Label: tbl_p202408_test
Timestamp: 2024-08-22-20-32-37
State: DOWNLOADING
RestoreObjs: {
  "name": "tbl_p202408_test",
  "database": "db",
  "olap_table_list": [
      {
          "name": "tbl",
          "partition_names": ["p202408"]
      }
]

### load to partition p202408, failed with exception
 curl --location-trusted -u root:"" \
>   -H "label:tbl_test_load_19" \
>   -H "timeout:300" \
>   -H "format: parquet" \
>   -T data_for_p202408.parquet \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3042,
    "Label": "tbl_test_load_19",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Table
[zt_order_detail_v3], Partition [p202408] is in restore process. Can not
load into it.etc.",
    "NumberTotalRows": 682,
    "NumberLoadedRows": 682,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 82025,
    "LoadTimeMs": 48,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 38,
    "CommitAndPublishTimeMs": 0
}

### load to partition p202408, successfully
$ curl --location-trusted -u root:"" \
>   -H "timeout:300" \
>   -H "format: json" \
>   -H "read_json_by_line:true" \
>   -T  data_for_p202407.json \
>   -XPUT http://fe_ip:8030/api/db/tbl/_stream_load
{
    "TxnId": 3043,
    "Label": "2f2dae38-a495-4c22-9492-419ea70b724e",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1128,
    "LoadTimeMs": 51,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 6,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 30,
    "CommitAndPublishTimeMs": 13
}

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/merged-1.0.1-deprecated PR has been merged into dev-1.0.1 kind/fix Categorizes issue or PR as related to a bug. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Restore with partition cause some query fail "table status is not NORMAL: RESTORE"

3 participants

@GoGoWen @morningman @jackwener