Oct 16, 2024: This week in DataFusion

[alamb]

Introduction

Goal of this ticket is a weekly summary if interesting things happening in DataFusion over the last week. Note this is not a complete list. Please feel free to comment on this ticket about things that I may have missed or you think should get wider attention by the community

Loosely inspired by https://this-week-in-rust.org/

Andrew's TLDR:

We are preparing for the 43.0.0 release and I am personally pretty excited about:

• [DISCUSSION] Make DataFusion the fastest engine for querying parquet data in ClickBench #12821
• [Epic] Unify WindowFunction Interface (remove built in list of BuiltInWindowFunction s) #8709
• [EPIC] Automatically generate all function documentation from code #12740

Upcoming Releases

• Release DataFusion 42.1.0 #12813 (thanks @Xuanwo and @matthewmturner)
• Release DataFusion 43.0.0 #12470 (thanks @andygrove)

Project Happenings

• Integrate sqlparser into DataFusion governance: [DISCUSSION]: move sqlparser to Apache (DataFusion) governance datafusion-sqlparser-rs#1294 (comment)

Highlights from last week(s):

(I am sorry if I missed you -- please add a note to this ticket with anything you would like to add)

• @dmitrybugakov started managing extension functions in https://github.com/datafusion-contrib/datafusion-functions-extra
• @eejbyfeldt is doing some great work on grouping sets such as feat: Implement grouping function using grouping id #12704
• @tokoko, @Blizzara @vbarua and @westonpace continue to mature the substrait support such as fix(substrait): remove optimize calls from substrait consumer #12800
• Along with @devanbenz and @Rachelint and @jayzhan211 I implemented Implement special min/max accumulator for Strings and Binary (10% faster for Clickbench Q28) #12792 to help clickbench queries
• @timesaucer made a beautiful macro Macro for creating record batch from literal slice #12846
• @Rachelint made a beautiful aggregation fuzzing proect
• @jonahgao continues to make our SQL handling more beautiful and correct (Remove unused dependencies and features #12808, Fix clippy error on wasmtest #12844, etc)

Performance

• [DISCUSSION] Make DataFusion the fastest engine for querying parquet data in ClickBench #12821 (thanks to the epic work of @Rachelint, @goldmedal, @jayzhan211, @Dandandan @XiangpengHao and others, we are quite close)
• [EPIC] Improvements to GroupColumn multi-column aggregation performance #12680 (kudos to @jayzhan211 and @Rachelint)
• @simonvandel and @tlm365 Optimize signum function (3-25x faster) #12890

Quality

• Aggregation fuzz testing #12114 (already found several bugs -- thanks @Rachelint)

Extensibility

• Very close to finishing [Epic] Unify WindowFunction Interface (remove built in list of BuiltInWindowFunction s) #8709 (thanks @jcsherin @jatin510 @hailelagi)
• @Omega359 started [EPIC] Automatically generate all function documentation from code #12740 and we are making great progress thanks to @jonathanc-n @juroberttyb and others
• @notfillipo and @findepi are working to better separate logical and physical types [EPIC] Decouple logical from physical types #12622

Features

Interesting discussions underway:

• 2024 Q3-Q4 Roadmap? #11442
• [DISCUSS] Document criteria for adding new features / what belongs in core DataFusion (e.g. sql syntax, functions, etc) #12357

Community

• Weekly Call
• Slack/Discord: info links

Upcoming meetups:

• Oct 14 Seattle: https://lu.ma/tnwl866b @phillipleblanc @likekim
• Dec 18 Chicago: https://lu.ma/eq5myc5i @adriangb @timsaucer

Background:

I got some great feedback from @timsaucer, @findepi and @andygrove on the DataFusion weekly call that having a weekly summary like #12494 was helpful. I will therefore try to write up one each week

[jonathanc-n]

@alamb I really like this, keeping one up each week would be great. Gives everybody a good direction to go in for the overall project. Thanks for writing this!

[alamb]

A discussion about meetup in Amsterdam:

• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 #12988

[alamb]

Something else I hope to highlight next week is how the process of reviewing PRs helps understand the code, helps the community, and drives the process forward

[alamb]

Next week's issue: #13035