Is your feature request related to a problem or challenge?
DataFusion provides extensibility for functions (UDFs/UDAFs/UDWFs) and expressions (ExprPlanner), but lacks a clean way to customize how SQL table factors (FROM clause elements) are planned into logical plans. This makes it difficult to implement:
Table-valued functions like PostgreSQL's generate_series() or SQL Server's OPENJSON() TableFunction actually covers this use case- Special SQL syntax currently unsupported by DataFusion like
PIVOT/UNPIVOT, TABLESAMPLE, or MATCH_RECOGNIZE
Existing workaround (see #17633) has a significant limitation in that it only work at query root: custom relations inside JOINs, CTEs, or subqueries aren't transformed.
Describe the solution you'd like
A RelationPlanner trait that intercepts table factor planning at any nesting level:
pub trait RelationPlanner: Debug + Send + Sync {
fn plan_relation(
&self,
relation: TableFactor,
context: &mut dyn RelationPlannerContext,
) -> Result<RelationPlanning>;
}Key features:
- Works for nested relations (JOINs, CTEs, subqueries)
- Chain multiple planners with
SessionContext::register_relation_planner()
- Access SQL utilities via
RelationPlannerContext (expression conversion, identifier normalization)
- Return
Planned(plan) to handle, or Original(relation) to pass through
Example usage:
ctx.register_relation_planner(Arc::new(MatchRecognizePlanner))?;
ctx.sql("SELECT * FROM stocks MATCH_RECOGNIZE(...);").await?;Describe alternatives you've considered
Additional context
Related to:
This complements existing extensibility APIs:
ExprPlanner: custom operators/expressionsTypePlanner: custom data typesRelationPlanner (new): custom table factors
Is your feature request related to a problem or challenge?
MATCH_RECOGNIZEfor Advanced Pattern Matching #13583DataFusion provides extensibility for functions (UDFs/UDAFs/UDWFs) and expressions (
ExprPlanner), but lacks a clean way to customize how SQL table factors (FROM clause elements) are planned into logical plans. This makes it difficult to implement:Table-valued functions like PostgreSQL'sgenerate_series()or SQL Server'sOPENJSON()TableFunctionactually covers this use casePIVOT/UNPIVOT,TABLESAMPLE, orMATCH_RECOGNIZEExisting workaround (see #17633) has a significant limitation in that it only work at query root: custom relations inside JOINs, CTEs, or subqueries aren't transformed.
Describe the solution you'd like
A
RelationPlannertrait that intercepts table factor planning at any nesting level:Key features:
SessionContext::register_relation_planner()RelationPlannerContext(expression conversion, identifier normalization)Planned(plan)to handle, orOriginal(relation)to pass throughExample usage:
Describe alternatives you've considered
SAMPLEusing extension APIs #17633: CustomQueryPlannerapproach, but only transforms root-level relationsAdditional context
Related to:
SAMPLEusing extension APIs #17633 - Attempting TABLESAMPLE via extensions (limited to root relations)This complements existing extensibility APIs:
ExprPlanner: custom operators/expressionsTypePlanner: custom data typesRelationPlanner(new): custom table factors