The Parquet encryption feature added in #16351 requires specifying AES keys for footer and column encryption directly. This is quite limiting as it assumes all Parquet files in a table use the same keys. It also makes it difficult to implement envelope encryption, where data encryption keys are encrypted with a KMS and the encrypted keys are stored in the Parquet metadata.
DataFusion should provide a more flexible way to configure Parquet encryption and allow file encryption and decryption properties to be generated per-file, and support using a KeyRetriever.
There is some previous discussion about this in #15216
The Parquet encryption feature added in #16351 requires specifying AES keys for footer and column encryption directly. This is quite limiting as it assumes all Parquet files in a table use the same keys. It also makes it difficult to implement envelope encryption, where data encryption keys are encrypted with a KMS and the encrypted keys are stored in the Parquet metadata.
DataFusion should provide a more flexible way to configure Parquet encryption and allow file encryption and decryption properties to be generated per-file, and support using a KeyRetriever.
There is some previous discussion about this in #15216