allow writing compressed files

[oridag]

Is your feature request related to a problem or challenge?

It would be useful to allow stream compression for output file formats that don't have built in compression spec like parquet

Describe the solution you'd like

DataFrame::write_json and DataFrame::write_csv can take an optional Compression argument

Describe alternatives you've considered

An alternative would be to post process the output files and compress them, but that's less efficient compared to stream compress them while writing

Additional context

No response

[alamb]

I suggest we add some sort of CsvWriteOption structure to make the API more general. Perhaps something like

/// DataFusion specific writing properties
struct WriteProperties {
  /// underlying arrow csv writer properties
  csv: CsvWriterProperties,
  compression: Compression
}

pub async fn write_csv(self, path: &str, writer_properties: Option<WriteProperties>) -> Result<()>

This would be consistent with the parquet write API: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.write_parquet

[jiangzhx]

I suggest we add some sort of CsvWriteOption structure to make the API more general. Perhaps something like

/// DataFusion specific writing properties
struct WriteProperties {
  /// underlying arrow csv writer properties
  csv: CsvWriterProperties,
  compression: Compression
}

pub async fn write_csv(self, path: &str, writer_properties: Option<WriteProperties>) -> Result<()>

This would be consistent with the parquet write API: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.write_parquet

@oridag have you working on this?
if not ,i will give this a try.

[oridag]

@jiangzhx i'm not, go ahead

[jiangzhx]

After doing some research, I found that PR #6526 allows you to write compressed data to a CSV file. However, currently it only works with SQL and not with DataFrames.

https://github.com/apache/arrow-datafusion/blob/36292f63e3c7dfbd5bca77f5e36c2aa43048d2be/datafusion/core/src/datasource/file_format/csv.rs#L473-L521

example:

CREATE EXTERNAL TABLE test (
        c1  TINYINT NOT NULL,
        c2  SMALLINT NOT NULL,

    )
STORED AS CSV
WITH HEADER ROW
COMPRESSION TYPE GZIP
LOCATION '/Users/sylar/workspace/opensource/arrow-datafusion/testing/data/csv/test.csv.gz';

INSERT INTO test select * from test;

[alamb]

After doing some research, I found that PR #6526 allows you to write compressed data to a CSV file. However, currently it only works with SQL and not with DataFrames.

The more we can unify the SQL / DataFrame codepaths the better. Since SQL has just started supporting writes, there is likely some non trivial unification that would be helpful to do

[devinjdangelo]

This is supported today in both SQL (see write_options) and DataFrames (see DataFrameWriteOptions parameter).

@alamb I believe we can close this issue as complete.

[alamb]

Thanks for the ping @devinjdangelo