ARROW-10783: [Rust][DataFusion] Implement Statistics for Parquet TableProvider#8944
ARROW-10783: [Rust][DataFusion] Implement Statistics for Parquet TableProvider#8944seddonm1 wants to merge 4 commits intoapache:masterfrom
Conversation
| let schema = parquet_exec.schema(); | ||
|
|
||
| let metadata = parquet_exec.metadata(); | ||
| let (num_rows, total_byte_size) = metadata.row_groups().iter().fold( |
There was a problem hiding this comment.
I think this would be better as two "loops", than it can use .sum() as well instead of fold?
There was a problem hiding this comment.
updated to two iterators
Dandandan
left a comment
Dandandan
left a comment
There was a problem hiding this comment.
LGTM, one style suggestion
| let total_byte_size = metadata | ||
| .row_groups() | ||
| .iter() | ||
| .map(|rg| rg.total_byte_size() as usize) |
There was a problem hiding this comment.
I think the as usize could move to below (Some(num_rows as usize)).
Then it could use .map(RowGroupMetaData::total_byte_size) as well?
There was a problem hiding this comment.
I am not familiar with this .map(RowGroupMetaData::total_byte_size) convention. do you know what this is called so I can look up the Rust docs?
There was a problem hiding this comment.
The way it is now is called "closure" and it could be replaced by a the name of the function instead. There is something on the clippy docs:
https://rust-lang.github.io/rust-clippy/master/#redundant_closure
There was a problem hiding this comment.
Thanks. I have updated the PR. I should have recognised this from Scala.
Codecov Report
@@ Coverage Diff @@
## master #8944 +/- ##
==========================================
- Coverage 83.26% 83.26% -0.01%
==========================================
Files 195 196 +1
Lines 48066 48131 +65
==========================================
+ Hits 40024 40078 +54
- Misses 8042 8053 +11
Continue to review full report at Codecov.
|
|
FYI @XiaokunDing |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #8944 +/- ##
==========================================
- Coverage 83.26% 83.26% -0.01%
==========================================
Files 195 196 +1
Lines 48066 48131 +65
==========================================
+ Hits 40024 40078 +54
- Misses 8042 8053 +11 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
6 participants
This is an implementation of Statistics for the Parquet datasource as this PR seems to have gone quiet: #8860
This PR exposes the ParquetMetaData produced by the file reader as even though initially we only consume
row_countandtotal_byte_sizethere is column level metadata that may be very useful in future for potentially large performance gains.