[fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down by seawinde · Pull Request #43539 · apache/doris

seawinde · 2024-11-09T13:17:20Z

What problem does this PR solve?

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this leftExtendedNodes Shoule be contained both orders and lineitem, which should be 3, but now is 2

Issue Number: close #xxx

Related PR: #28006

Problem Summary:

This may cause associate wrongly when use hyper graph and materialized view which depends on it rewrite wrongly

Release note

Fix generate hyper graph wrongly when has filter which can not push down

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…n not push down

doris-robot · 2024-11-09T13:17:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2024-11-09T13:24:38Z

run buildall

seawinde · 2024-11-11T06:47:48Z

run buildall

doris-robot · 2024-11-11T08:08:24Z

TPC-H: Total hot run time: 52043 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0cecf414730a5235248f752ef16ed3c0d8ef709, data reload: false

------ Round 1 ----------------------------------
q1	17604	7529	7303	7303
q2	2583	1310	1305	1305
q3	9954	1155	1134	1134
q4	10228	870	861	861
q5	7604	3018	2971	2971
q6	228	144	145	144
q7	1031	606	604	604
q8	9394	2365	2378	2365
q9	12727	12515	12593	12515
q10	7151	2430	2425	2425
q11	457	259	257	257
q12	404	212	211	211
q13	17798	3036	3028	3028
q14	252	208	214	208
q15	590	531	508	508
q16	691	583	576	576
q17	997	520	577	520
q18	7242	6710	6792	6710
q19	1324	983	1170	983
q20	3148	2912	2826	2826
q21	4018	3210	3316	3210
q22	1409	1379	1380	1379
Total cold run time: 116834 ms
Total hot run time: 52043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7516	7490	7436	7436
q2	339	244	250	244
q3	3112	2947	2976	2947
q4	2113	1850	1838	1838
q5	5691	5663	5716	5663
q6	223	138	137	137
q7	2224	1822	1831	1822
q8	3388	3551	3578	3551
q9	14774	14731	14607	14607
q10	3652	3614	3548	3548
q11	596	491	505	491
q12	857	607	605	605
q13	13696	3252	3282	3252
q14	315	259	278	259
q15	585	533	536	533
q16	665	624	639	624
q17	1833	1655	1623	1623
q18	8243	7806	7655	7655
q19	1690	1568	1485	1485
q20	2183	1932	1965	1932
q21	5293	5306	5174	5174
q22	637	558	550	550
Total cold run time: 79625 ms
Total hot run time: 65976 ms

morrySnow · 2024-11-13T06:48:39Z

run performance

github-actions · 2024-11-13T11:18:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-13T11:18:42Z

PR approved by anyone and no changes requested.

xzj7019 · 2024-11-13T11:37:19Z

            Set<Expression> queryPredicates,
            SlotMapping queryToViewMapping,
+            StructInfo queryStructInfo,
+            StructInfo viwStructInfo,


viwStructInfo -> viewStructInfo

seawinde · 2024-11-13T13:42:45Z

run buildall

github-actions · 2024-11-14T06:25:15Z

PR approved by at least one committer and no changes requested.

…n not push down (#43539) ### What problem does this PR solve? Related PR: #28006 Problem Summary: When query is as following: select o_orderdate, o_shippriority, o_comment, l_orderkey, l_partkey from orders left join lineitem on l_orderkey = o_orderkey left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey; hyper graph is as following: this` leftExtendedNodes` Shoule be contained both orders and lineitem, which should be 3, but now is 2 This may cause associate wrongly when use hyper graph and materialized view which depends on it rewrite wrongly ### Release note Fix generate hyper graph wrongly when has filter which can not push down

…ter which can not push down #43539 (#43944) Cherry-picked from #43539 Co-authored-by: seawinde <wusi@selectdb.com>

…ter which can not push down #43539 (#43946) Cherry-picked from #43539 Co-authored-by: seawinde <wusi@selectdb.com>

…h wrongly when has filter (apache#3649) pr: apache#43531 commitId: b2e42f2 pr: apache#43539 commitId: eb6c00c b0cecf4 --------- Co-authored-by: seawinde <wusi@selectdb.com>

[fix](mtmv) Fix generate hyper graph wrongly when has filter which ca…

eb6c00c

…n not push down

wm1581066 added usercase Important user case type label dev/2.1.x dev/3.0.x labels Nov 9, 2024

fix rewrite fail when has same col filter

b0cecf4

gavinchou added the cir label Nov 11, 2024

morrySnow previously approved these changes Nov 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

github-actions bot added the reviewed label Nov 13, 2024

xzj7019 reviewed Nov 13, 2024

View reviewed changes

xzj7019 approved these changes Nov 13, 2024

View reviewed changes

fix regression comment

b950f0c

seawinde dismissed morrySnow’s stale review via b950f0c November 13, 2024 13:42

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

morrySnow approved these changes Nov 14, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 14, 2024

morrySnow merged commit 182af8e into apache:master Nov 14, 2024

github-actions bot mentioned this pull request Nov 14, 2024

branch-3.0: [fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down #43539 #43944

Merged

github-actions bot mentioned this pull request Nov 14, 2024

branch-2.1: [fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down #43539 #43946

Merged

dataroaring pushed a commit that referenced this pull request Nov 14, 2024

branch-3.0: [fix](mtmv) Fix generate hyper graph wrongly when has fil…

b3a1025

…ter which can not push down #43539 (#43944) Cherry-picked from #43539 Co-authored-by: seawinde <wusi@selectdb.com>

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Nov 14, 2024

yiguolei pushed a commit that referenced this pull request Nov 16, 2024

branch-2.1: [fix](mtmv) Fix generate hyper graph wrongly when has fil…

91dcc0e

…ter which can not push down #43539 (#43946) Cherry-picked from #43539 Co-authored-by: seawinde <wusi@selectdb.com>

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Nov 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down#43539

[fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down#43539
morrySnow merged 3 commits intoapache:masterfrom
seawinde:fix_filter_edge_in_hyper_graph_wrong

seawinde commented Nov 9, 2024 •

edited by morrySnow

Loading

Uh oh!

doris-robot commented Nov 9, 2024

Uh oh!

seawinde commented Nov 9, 2024

Uh oh!

seawinde commented Nov 11, 2024

Uh oh!

doris-robot commented Nov 11, 2024

Uh oh!

morrySnow commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

xzj7019 Nov 13, 2024

Uh oh!

seawinde commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

seawinde commented Nov 9, 2024 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 9, 2024

Uh oh!

seawinde commented Nov 9, 2024

Uh oh!

seawinde commented Nov 11, 2024

Uh oh!

doris-robot commented Nov 11, 2024

Uh oh!

morrySnow commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

xzj7019 Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

seawinde commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

seawinde commented Nov 9, 2024 •

edited by morrySnow

Loading