Skip to content

[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions#23874

Merged
JNSimba merged 1 commit intoapache:masterfrom
CalvinKirs:streamload-where-cause
Sep 5, 2023
Merged

[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions#23874
JNSimba merged 1 commit intoapache:masterfrom
CalvinKirs:streamload-where-cause

Conversation

@CalvinKirs
Copy link
Copy Markdown
Member

@CalvinKirs CalvinKirs commented Sep 4, 2023

Steps to reproduce：

 curl --location-trusted  -H "Expect: 100-continue" -X PUT  -u root:  -H "format:json" -H "where:d = 'doris' or d = 'nb'"  -d '{"a":"3","b":"2","c":"389","d":"nb"}' http://localhost:8030/api/kris/test/_stream_load

java.lang.IllegalStateException: null
        at com.google.common.base.Preconditions.checkState(Preconditions.java:492) ~[guava-30.0-jre.jar:?]
        at org.apache.doris.analysis.SlotRef.getTableName(SlotRef.java:264) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.rewriteOrToIn(ExtractCommonFactorsRule.java:532) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.makeCompoundRemaining(ExtractCommonFactorsRule.java:455) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.extractCommonFactors(ExtractCommonFactorsRule.java:205) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.apply(ExtractCommonFactorsRule.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExprRewriter.applyRuleOnce(ExprRewriter.java:146) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExprRewriter.rewrite(ExprRewriter.java:139) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initWhereExpr(ExternalFileScanNode.java:312) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initAndSetWhereExpr(ExternalFileScanNode.java:293) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initParamCreateContexts(ExternalFileScanNode.java:274) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.init(ExternalFileScanNode.java:207) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.StreamLoadPlanner.plan(StreamLoadPlanner.java:201) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.load.routineload.RoutineLoadJob.plan(RoutineLoadJob.java:804) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.load.routineload.KafkaTaskInfo.rePlan(KafkaTaskInfo.java:117) ~[doris-fe.jar:1.2-SNAPSHOT]

@CalvinKirs
Copy link
Copy Markdown
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

1 similar comment
@CalvinKirs
Copy link
Copy Markdown
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.12 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161949833 Bytes

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.66 seconds
stream load tsv: 532 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162029829 Bytes

@CalvinKirs
Copy link
Copy Markdown
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.18 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162054916 Bytes

@CalvinKirs CalvinKirs force-pushed the streamload-where-cause branch from e11a76f to 8e35f43 Compare September 5, 2023 01:29
@CalvinKirs
Copy link
Copy Markdown
Member Author

CalvinKirs commented Sep 5, 2023

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Sep 5, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.67 seconds
stream load tsv: 534 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162194766 Bytes

dataroaring
dataroaring approved these changes Sep 5, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2023
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 5, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 5, 2023

PR approved by anyone and no changes requested.

JNSimba
JNSimba approved these changes Sep 5, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit 4dac2d3 into apache:master Sep 5, 2023
@CalvinKirs CalvinKirs deleted the streamload-where-cause branch September 5, 2023 03:28
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 1, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
2495a1e 
…contains complex where conditions apache#23874
@CalvinKirs CalvinKirs mentioned this pull request Nov 1, 2023
Merged
xiaokang pushed a commit that referenced this pull request Nov 1, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
a321f63 
…contains complex where conditions #23874 (#26212)
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
fb6182c 
…contains complex where conditions apache#23874 (apache#26212)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JNSimba JNSimba JNSimba approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/1.2.9 dev/2.0.3-merged merge_conflict reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@CalvinKirs @doris-robot @JNSimba @dataroaring @xiaokang @morningman