Make a faster way to check column existence in optimizer (not `is_err()`)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Related to #5157

There are many places in the code that use fallible functions on DFSchema to check if a column exists:
https://docs.rs/datafusion-common/18.0.0/datafusion_common/struct.DFSchema.html#method.index_of
https://docs.rs/datafusion-common/18.0.0/datafusion_common/struct.DFSchema.html#method.index_of_column_by_name
https://docs.rs/datafusion-common/18.0.0/datafusion_common/struct.DFSchema.html#method.field_from_column

For example, there is code that looks like this (call is_ok() or is_err()and totally discards the error with the string)

input_schema.field_from_column(col).is_ok()

This is problematic because they return a DataFusionError that not only has an allocated String but also often has gone through a lot of effort to construct a nice error message. You can see them appearing in the trace on #5157

As part of making the optimizer faster Related to #5157 we need to avoid these string allocations,

Thus I propose:

1. Add new functions for checking that return a bool rather than an error
2. Replace the use of is_err() with

Find the field with the given qualified column

For example,

impl DFSchema {
  // existing function that returns Result
  pub fn field_from_column(&self, column: &Column) -> Result<&DFField> {...}

  // new function that returns bool  <---- Add this new function
  pub fn has_column(&self, column: &Column) -> bool {...}
}

And then replace in the code that have the pattern

input_schema.field_from_column(col).is_ok()

With

input_schema.has_column(col)

Describe the solution you'd like
Ideally someone would do this transition one function on DFSchema at a time (not one giant PR please 🙏 )

Describe alternatives you've considered
There are more involved proposals for larger changes to DFSchema but simply avoiding this check might help a lot

Additional context
I think this is a good first exercise as the desire is well spelled out and it is a software engineering exercise rather than requires deep datafusion expertise

[suxiaogang223]

i'm happy to do this🌝

[ygf11]

Seems it is relative to the pr #5287.

[suxiaogang223]

Seems it is relative to the pr #5287.

Yes, I think the method had_column also should distinguish FieldNotFound and Ambiguous reference error.
Maybe the new method should be

pub fn has_column(&self, column: &Column) -> Result<bool> {...}

[alamb]

The key for performance is not to return a DataFusionError with a allocated string

[suxiaogang223]

The key for performance is not to return a DataFusionError with a allocated string

Maybe we can use assert to assume that the "Ambiguous reference error" should not happen in had_column?

[alamb]

Maybe we can use assert to assume that the "Ambiguous reference error" should not happen in had_column?

How about returning an enum like

enum FoundColumn {
  Found,
  NotFound,
  Ambiguous
}


pub fn has_column(&self, column: &Column) -> FoundColumn {...}

?

[suxiaogang223]

I think returning enum will be same as returning result, because the caller also have to handle Ambiguous and return an Err.

Returning Result<bool> can also avoid str allocating in field_from_name().is_err(). The code will be like this:

if schema.had_column(col)? {...}

Maybe the key question is that do we need to check 'Ambiguous error' each time the had_column called? Actually we can just check Ambiguous err once at begin.

I'm not sure if my thinking is correct, need your advice @alamb 🤓

[alamb]

Not sure -- will try and check out #5328 shortly