[EPIC] DuckDB-Inspired Feature Enhancements

[berkaysynnada]

While I'm investigating some possible project ideas for GSoC 2025, I took a look at some cool DuckDB features and want to create issues for similar enhancements in datafusion. These features will improve datafusion's usability and performance.

• Implement UNION ALL BY NAME #14508
• Add Native Support for List Functions (Dependent on Lambda Support) #14509
  (Note: Lambda support is tracked in #14205.)
• Add Memory Profiling Functionality #14510
• Add Support for Dynamic SQL Macros for Flexible Column Selection #14512
• Implement Nicer / DuckDB style explain plans #9371 (This was issued long ago, and also directly linked to GSoC project proposals)

There are also smaller-scale, but still very cool, features in this list:
https://duckdb.org/docs/sql/dialect/friendly_sql.html

Anyone is welcome to undertake any missing one from that list.

[Omega359]

#9371

[PokIsemaine]

Hello @berkaysynnada ,

I am interested in both datafusion and DuckDB and would like to participate in this year's GSoC.

I haven't deeply explored the core code of datafusion yet, so I would like to ask if you could provide references for similar functionality implementations, such as past issues, pull requests, discussions, or blog posts. These resources would help me get familiar with the project more quickly.

If there are any suitable good-first-issues that could help me understand the relevant modules, I would be happy to try solving them.

[berkaysynnada]

Hello @PokIsemaine. I have an idea that hasn't been issued yet: implementing "ORDER BY ALL." Although this feature is not part of standard sql, duckdb supports it https://duckdb.org/docs/sql/query_syntax/orderby.html#order-by-all

If you're looking for a warm-up project, implementing it in datafusion would be a great start. I suggest familiarizing yourself with the various parts of the codebase rather than diving straight into the core features. For guidance, you might want to look at how "group by all" is handled.

[PokIsemaine]

Hello @PokIsemaine. I have an idea that hasn't been issued yet: implementing "ORDER BY ALL." Although this feature is not part of standard sql, duckdb supports it https://duckdb.org/docs/sql/query_syntax/orderby.html#order-by-all

If you're looking for a warm-up project, implementing it in datafusion would be a great start. I suggest familiarizing yourself with the various parts of the codebase rather than diving straight into the core features. For guidance, you might want to look at how "group by all" is handled.

Thanks for your advice!

I noticed #7622, if the syntax was originally not supported by sqlparser, do I need to first submit a PR in the sqlparser repository and then proceed with the implementation in datafusion? Or should we just go ahead with the implementation in datafusion?

[berkaysynnada]

I noticed #7622, if the syntax was originally not supported by sqlparser, do I need to first submit a PR in the sqlparser repository and then proceed with the implementation in datafusion? Or should we just go ahead with the implementation in datafusion?

Yes, you probably need to update struct OrderBy in sqlparser

[PokIsemaine]

I noticed #7622, if the syntax was originally not supported by sqlparser, do I need to first submit a PR in the sqlparser repository and then proceed with the implementation in datafusion? Or should we just go ahead with the implementation in datafusion?

Yes, you probably need to update struct OrderBy in sqlparser

apache/datafusion-sqlparser-rs#1724
I updated OrderBy in sqlparser and made it support parsing of ORDER BY ALL, after the subsequent version is released, we can start implementing ORDER BY ALL for datafusion

[berkaysynnada]

@PokIsemaine Awesome! I have not even noticed yet you open the sqlparser PR 😅 I hope it won't take much time to release the new version.

[2010YOUY01]

DuckDB can configure memory limit through SQL interface, here is a related issue to enable it in DataFusion #15552