Make data type of FileScanConfig.table_partition_cols a vector of Field

[NGA-TRAN]

Is your feature request related to a problem or challenge?

Currently, the table_partition_cols of FileScanConfig is a vector of (String, DataType) to store only column name and data type https://github.com/apache/arrow-datafusion/blob/37d6bf08c948418fe6c72d072d988c2875d81e02/datafusion/core/src/datasource/physical_plan/file_scan_config.rs#L104

In our IOx case, a column has more meaning than just name and data type and need the Field data type to convey that. We hit a bug of missing column schema information because of missing other information a Field provides

Describe the solution you'd like

Replace pub table_partition_cols: Vec<(String, DataType)>,
with pub table_partition_cols: Vec<Field>,

Describe alternatives you've considered

No response

Additional context

No response

[NGA-TRAN]

@alamb
If there is no objection, I will start working on this

[crepererum]

That sounds reasonable to me.

[alamb]

It is reasonable in theory, however, this will likely be a non trivial API change, so we should take significant care needs to be taken on the API to make migration as easy as possible (for example, a function that takes the old data type and makes a new one), or maybe a with_partition_columns type function

[NGA-TRAN]

It is reasonable in theory, however, this will likely be a non trivial API change, so we should take significant care needs to be taken on the API to make migration as easy as possible (for example, a function that takes the old data type and makes a new one), or maybe a with_partition_columns type function

Agreed. I will keep the current API and add one more to make sure it still works for current folks

[NGA-TRAN]

I start working on this today