Is your feature request related to a problem or challenge?
When pushdown_filters is enabled, DF should be able to eliminate the subsequent filter.
When enabling the option for tpc-h benchmark, the FilterExec and l_shipdate projection is still present in the plans.
For example query 3 we can see the filter:
FilterExec: l_shipdate@3 > 9204
ParquetExec: file_groups={2 groups: [[...]]},
projection=[l_orderkey, l_extendedprice, l_discount, l_shipdate], predicate=l_shipdate@10 > 9204, pruning_predicate=l_shipdate_max@0 > 9204
Describe the solution you'd like
Remove the filter when.
We probably need to make some changes to TableProvider FileFormat to support removing the filter based on the file format.
Describe alternatives you've considered
No response
Additional context
No response
Is your feature request related to a problem or challenge?
When
pushdown_filtersis enabled, DF should be able to eliminate the subsequent filter.When enabling the option for tpc-h benchmark, the
FilterExecandl_shipdateprojection is still present in the plans.For example query 3 we can see the filter:
Describe the solution you'd like
Remove the filter when.
We probably need to make some changes to
TableProviderFileFormatto support removing the filter based on the file format.Describe alternatives you've considered
No response
Additional context
No response