[EPIC] Add support for all join operators

[viirya]

This is an umbrella ticket for adding Join support to Comet. In Spark, there are basically three types of Join operators: BroadcastJoin, HashJoin, SortMergeJoin. In DataFusion, two Join operators are supported: HashJoin, SortMergeJoin (experimental).

We are going to delegate Spark Join operators to correspond DataFusion Join operators. We will also go to improve DataFusion Join operators if needed.

SortMergeJoin

• Relax join keys constraint from Column to any physical expression for physical join operators Relax join keys constraint from Column to any physical expression for physical join operators datafusion#8991
• Support join filter for SortMergeJoin Support join filter for SortMergeJoin datafusion#9080
• Add spilling in SortMergeJoin Add spilling in SortMergeJoin datafusion#9359
• Support SortMergeJoin operator - Support sort merge join #177
  ...

HashJoin

• Support HashJoin operator - Support HashJoin operator #193

BroadcastJoin

• Add CometBroadcastExchangeExec support broadcasting the result of Comet native operator - Add CometBroadcastExchangeExec to support broadcasting the result of Comet native operator #81
• Support BroadcastHashJoinExec Support BroadcastHashJoinExec #202
• Enable Comet broadcast by default Enable Comet broadcast by default #212

Other Join operators (Cross Join)

• Support CartesianProductExec - Support CartesianProductExec #199
• Support BroadcastNestedLoopJoinExec - Support BroadcastNestedLoopJoinExec #198

[mwiewior]

Hey @viirya - it's really awesome ! Do you happen to know when approx these BHJ and shuffle components will be open sourced?

[viirya]

Hi @mwiewior

I will open source shuffle components soon. For join support, it is in development now. We already have working code but get some test failure on Spark tests. Once these test failures are resolved, we will open source it.

[viirya]

For shuffle, we just remove it from the initial open source code. So the code is ready, we only need to add it back to Comet.

[mwiewior]

Great news ! Thx !