Is your feature request related to a problem or challenge? Please describe what you are trying to do.
RecordBatchStream is a DataFusion type. However it carries an ArrowError instead of DataFusionError. This leads to some wild error wrapping like this one:
https://github.com/apache/arrow-datafusion/blob/5883e43db6c16d3ac3616302606849abbfbc86eb/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L181
This makes it hard to detect the actual underlying error, e.g. ResourceExhausted for "out of memory" (also see #3941 where this would be useful).
Describe the solution you'd like
Make RecordBatchStream a DataFusionError instead. I'm aware this is a quite major change.
Describe alternatives you've considered
-
Additional context
-
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
RecordBatchStreamis a DataFusion type. However it carries anArrowErrorinstead ofDataFusionError. This leads to some wild error wrapping like this one:https://github.com/apache/arrow-datafusion/blob/5883e43db6c16d3ac3616302606849abbfbc86eb/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L181
This makes it hard to detect the actual underlying error, e.g.
ResourceExhaustedfor "out of memory" (also see #3941 where this would be useful).Describe the solution you'd like
Make
RecordBatchStreamaDataFusionErrorinstead. I'm aware this is a quite major change.Describe alternatives you've considered
-
Additional context
-