[Parquet][Python] API to decrypt parquet file using one DEK and no metadata

[changhu-m]

Describe the enhancement requested

I am having trouble decrypting a parquet file encrypted in the C++ Arrow library using PyArrow.

In C++, I encrypt the file with one key and no metadata.

  // Convert vector<uint8_t> to string for the FileEncryptionProperties::Builder
  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  if (!result.ok()) {
    throw std::runtime_error("Failed to open output file: " + outputFilePath);
  }
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

I can decrypt the file in C++ using the same key and no metadata.

  const std::string keyStr(key.begin(), key.end());
  auto decryptionProps =
      parquet::FileDecryptionProperties::Builder().footer_key(keyStr)->build();
  auto readerProps = parquet::ReaderProperties();
  readerProps.file_decryption_properties(decryptionProps);

  std::unique_ptr<parquet::ParquetFileReader> reader =
      parquet::ParquetFileReader::OpenFile(outputPath, false, readerProps);

  // Read the file content into an arrow table actualTable using reader
  std::shared_ptr<arrow::Table> actualTable;
  std::unique_ptr<parquet::arrow::FileReader> arrowReader;
  auto status = parquet::arrow::FileReader::Make(
      arrow::default_memory_pool(), std::move(reader), &arrowReader);

  status = arrowReader->ReadTable(&actualTable);

However, I can't find an API in PyArrow to do the equivalent.

In Python, I can do the following, but it requires setting extra dummy key metadata in c++ (i.e., this code won't decrypt a file encrypted by the C++ approach at the beginning of this post).

import pyarrow.parquet.encryption as pe

# Create a simple KMS client that returns our DEK
class SimpleKmsClient(pe.KmsClient):
    def __init__(self):
        pe.KmsClient.__init__(self)
    
    def unwrap_key(self, wrapped_key, master_key_identifier):
        return dek
    
    def wrap_key(self, key_bytes, master_key_identifier):
        raise NotImplementedError("wrap_key not needed for decryption")

# Create KMS factory
def kms_factory(kms_connection_configuration):
    return SimpleKmsClient()

crypto_factory = pe.CryptoFactory(kms_factory)

# Simple decryption config
decryption_config = pe.DecryptionConfiguration()
kms_connection_config = pe.KmsConnectionConfig()

# Create file decryption properties
file_decryption_props = crypto_factory.file_decryption_properties(
    kms_connection_config, decryption_config
)

# Read the file
table = pq.read_table(path, decryption_properties=file_decryption_props)

My question is: Is there any Pyhton API to decrypt using only one DEK and no metadata?

Component(s)

Python

[raulcd]

cc @rok @EnricoMi

btw, @kou as discussed on the past should this be converted to a discussion into Q&A instead of an issue?

[rok]

We currently don't have a Python API where a single DEK could be used which is on purpose to encourage use of KMS over single keys (which is not recommended security-wise).
However users are asking for this and we should provide a good example of how to use a KMS in C++.
Alternatively we could provide a pe.NonSecureKMS in Python API that would provide what is asked for here but would be easy to swap out for a secure KMS when needed. I'd prefer the first option as it doesn't weaken our encryption.

[rok]

cc @adamreeve

[changhu-m]

Thanks for confirming, @rok.

I'd prefer the first option as it doesn't weaken our encryption.

Can you help me understand why a using a single DEK weaken the encryption? In my use case, we do encrypt the DEK and pass it in a secure channel. In a thought experiment, can users do exactly what the Arrow library does, but just do it external to the library and then pass in the key?

I am not a security expert, so please educate me.

[akoshelev]

I second @changhu-m question - it is not 100% clear how using externally managed keys can weaken the security story.

The use of KMS by default is a sensible choice. It seems that there is no other workaround for self-managed keys rather than creating an in-memory KMS and adding dummy metadata which is not ideal

[rok]

Can you help me understand why a using a single DEK weaken the encryption? In my use case, we do encrypt the DEK and pass it in a secure channel. In a thought experiment, can users do exactly what the Arrow library does, but just do it external to the library and then pass in the key?

Sorry, weakening encryption is not the correct expression here. We're encouraging use of KMSes as it helps users not having to re-implement things needed for envelope encryption, see design doc. I agree it would be nice to have more choices in pyarrow encryption story. We would want to be cautious to encourage good key management practices and make it easy to follow envelope encryption as per spec.

[adamreeve]

For a bit more context, here's the old mailing list discussion and issue where it was decided to only expose the higher level in Python.

The main arguments against exposing the low level in Python (from @ggershinsky ) were:

security: low-level encryption API is easy to misuse (eg giving the same keys for a number of different files; this'd break the AES GCM cipher). The high-level encryption layer handles that by applying envelope encryption and other best practices in data security. Also, this layer is maintained by the community, meaning that future improvements and security fixes can be upstreamed by anyone, and available to all.

compatibility: parquet-mr implements the high-level encryption layer. If we want the files produced by Spark/Presto/etc to be readable by pandas/PyArrow (and vice versa), we need to provide the Arrow users with the high-level API.

[rok]

Thanks for the context @adamreeve !

Perhaps a point of interest - we recently had an issue reported #47254 where encrypted readers/writers are not compatible due to one implementation avoiding KMS (I think) and not following the spec.

To be clear @changhu-m @akoshelev - PyArrow / arrow-cpp (and Parquet in general) encryption could use more examples / better examples and documentation (e.g. an example of how to use a real KMS with arrow-cpp and PyArrow). API change proposals are welcome too, we'd just want to discuss them before proceeding to implementation.

To not stifle progress here - @changhu-m could you say what your use case is and if you can see another way forward that doesn't include opening the low-level API in PyArrow?

[changhu-m]

Thanks @adamreeve @rok .

Our use case is the following:

Encryption side:

• A parquet file is encrypted with a DEK.
• The DEK is encrypted with KEK.
• The encrypted DEK and related metadata (i.e., KMS info to get KEK) is printed as JSON in a metadata file.
• The parquet file and metadata file are uploaded to a cloud location.

Our C++ code, as mentioned, is:

  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

Decryption side:

• Customers get the KEK using the metadata sent.
• Customers decrypt the DEK.
• Customer sets up parquet decryption with the DEK.

The Python code is:

import pyarrow.parquet.encryption as pe

# Create a simple KMS client that returns our DEK
class SimpleKmsClient(pe.KmsClient):
    def __init__(self):
        pe.KmsClient.__init__(self)
    
    def unwrap_key(self, wrapped_key, master_key_identifier):
        return dek
    
    def wrap_key(self, key_bytes, master_key_identifier):
        raise NotImplementedError("wrap_key not needed for decryption")

# Create KMS factory
def kms_factory(kms_connection_configuration):
    return SimpleKmsClient()

crypto_factory = pe.CryptoFactory(kms_factory)

# Simple decryption config
decryption_config = pe.DecryptionConfiguration()
kms_connection_config = pe.KmsConnectionConfig()

# Create file decryption properties
file_decryption_props = crypto_factory.file_decryption_properties(
    kms_connection_config, decryption_config
)

# Read the file
table = pq.read_table(path, decryption_properties=file_decryption_props)

And the Spark code is:

    
    # Create Spark session with encryption support
    spark = SparkSession.builder \
        .appName("ParquetModularEncryption") \
        .config('spark.jars', 'parquet-hadoop-1.13.1-tests.jar')\
        .config("spark.sql.adaptive.enabled", "true") \
        .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \
        .config("spark.hadoop.parquet.encryption.kms.client.class", "org.apache.parquet.crypto.keytools.mocks.InMemoryKMS") \
        .config("spark.hadoop.parquet.encryption.key.list", "k1:{DEK}") \
        .config("spark.hadoop.parquet.crypto.factory.class", "org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory") \
        .getOrCreate()


    # Read the encrypted parquet file (automatically decrypted with Hadoop config)
    df = spark.read.parquet(encrypted_path)

Right now, with the existing Python API, I have to set some dummy metadata on the encryption side to satisfy it, so the encryption code becomes:

  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  // Set up key metadata
  folly::dynamic metadata = folly::dynamic::object("keyMaterialType", "PKMT1")(
      "internalStorage", true)("isFooterKey", true)("doubleWrapping", false)(
      "kmsInstanceID", "dummy_kms_instance_id")(
      "kmsInstanceURL", "dummy_kms_instance_url")(
      "masterKeyID", "dummy_master_key_id")("wrappedDEK", "dummy_wrapped_dek");

  const std::string& footerKeyMetadata = folly::toJson(metadata);

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->footer_key_metadata(footerKeyMetadata)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

Because of this workaround, we are not blocked.

If we want to keep using table = pq.read_table(path, decryption_properties=file_decryption_props), is there a way to identify through decryption_properties that a DEK is already provided, and that no more metadata is required? Right now, metadata parsing is required before DEK retrieval.

[akoshelev]

To add to this, our interpretation of section 4.3 Parquet encryption specification is that for keys fully managed by the caller code, metadata section is not required

Key metadata can also be empty - in a case the encryption keys are fully managed by the caller code, and passed explicitly to Parquet readers for the file footer and each encrypted column.

So we were looking for ways to avoid providing dummy data for the purpose of using our own encryption keys

[adamreeve]

Above you describe your use case as:

• A parquet file is encrypted with a DEK.
• The DEK is encrypted with KEK.
• The encrypted DEK and related metadata (i.e., KMS info to get KEK) is printed as JSON in a metadata file.
• The parquet file and metadata file are uploaded to a cloud location.

This is exactly what the higher level API does for you if you set internal_key_material to false (although it also uses double wrapping by default, but you can disable that too). So I would suggest looking more closely at the high level API to see if it can work for you, rather than re-implement this yourself.

That said, I think it would be OK to expose the lower-level API in Python for use cases where users want more control over how keys are managed, as long as the downsides are clearly documented and users are directed to the higher level API by default (the C++ docs/examples could probably be improved here).

I believe PyArrow is the only Parquet implementation that supports encryption but doesn't expose the lower level API, so this would seem to actually hinder compatibility. For example, I'm not super familiar with parquet-java, but I think you can set spark.hadoop.parquet.crypto.factory.class to your own class that implements EncryptionPropertiesFactory and DecryptionPropertiesFactory, instead of having to use the PropertiesDrivenCryptoFactory.

[ggershinsky]

+1
In the old thread, my main concern was the lack of a full implementation of the high level API in PyArrow at the time. This would've led to incompatibility with Spark etc, and to a risk of producing files with broken encryption.
Now that the high level API is available for all developers, it would be fine to add an access to the low level APIs for expert use - with a proper documentation outlining the risks and pointing to the high level APIs

[rok]

We seem to have consensus on introducing some low-level API in Python. @changhu-m @akoshelev would internal_key_material set to false (Python would need to expose this too) plus addition of EncryptionPropertiesFactory and DecryptionPropertiesFactory solve your issue?