feat(auth): Implement python mtls helpers by nbayati · Pull Request #17495 · googleapis/google-cloud-python

nbayati · 2026-06-17T17:59:02Z

This PR builds on top of PR #16976. Once that PR is merged to main, this PR will be rebased and would only have the commit that introduces the helper methods (commit bc6d2b8), as the commits before this one are all from the other PR.

This PR provides helper methods to allow custom HTTP and WebSocket connection pools (such as those in google-genai and google-adk) to load default client certificates and resolve the GOOGLE_API_USE_MTLS_ENDPOINT env var. Changes include:

Introduced GOOGLE_API_USE_MTLS_ENDPOINT environment variable to control whether an mTLS endpoint should be used (always, never, or auto).
Added several new helper functions in google.auth.transport.mtls to facilitate SSL context creation and client certificate loading:
- load_client_cert_into_context: Loads a client certificate and key into a provided SSL context.
- make_client_cert_ssl_context: Creates a default SSL context loaded with a specific client certificate and key.
- load_default_client_cert: Discovers and loads the default client certificate into a provided SSL context if mTLS is enabled.
- get_default_ssl_context: Returns a default SSL context pre-loaded with the default client certificate, or None if unavailable.
- should_use_mtls_endpoint: Determines if an mTLS endpoint should be used based on the new environment variable and certificate availability.
Fixed outdated docstrings for default_client_cert_source and default_client_encrypted_cert_source to correctly state they raise MutualTLSChannelError instead of DefaultClientCertSourceError.
Updated default_client_cert_source to also catch ClientCertError when loading credentials.
Added comprehensive unit tests for the new mTLS helper methods.

Replace pyOpenSSL with standard library ssl for mTLS transport and update key decryption to use cryptography library. This change also enhances security for handling private keys by: - Using Linux memfd_create for RAM-backed in-memory files to avoid writing secrets to physical storage. - Encrypting plaintext keys on-the-fly before writing to fallback temporary files on disk. - Securely wiping temporary files with null bytes before deletion.

…S cert loading fallbacks

… update tests accordingly

…on to include empty password for client_combined_cert_path

…ndwritten SDK mTLS support - Introduced `GOOGLE_API_USE_MTLS_ENDPOINT` environment variable to control whether an mTLS endpoint should be used (`always`, `never`, or `auto`). - Added several new helper functions in `google.auth.transport.mtls` to facilitate SSL context creation and client certificate loading: - `load_client_cert_into_context`: Loads a client certificate and key into a provided SSL context. - `make_client_cert_ssl_context`: Creates a default SSL context loaded with a specific client certificate and key. - `load_default_client_cert`: Discovers and loads the default client certificate into a provided SSL context if mTLS is enabled. - `get_default_ssl_context`: Returns a default SSL context pre-loaded with the default client certificate, or `None` if unavailable. - `should_use_mtls_endpoint`: Determines if an mTLS endpoint should be used based on the new environment variable and certificate availability. - Fixed outdated docstrings for `default_client_cert_source` and `default_client_encrypted_cert_source` to correctly state they raise `MutualTLSChannelError` instead of `DefaultClientCertSourceError`. - Updated `default_client_cert_source` to also catch `ClientCertError` when loading credentials. - Added comprehensive unit tests for the new mTLS helper methods.

gemini-code-assist

Code Review

This pull request removes the dependency on pyOpenSSL and cffi across the codebase, transitioning to the standard library ssl module and the cryptography library for mutual TLS (mTLS) support. It introduces a secure three-tier fallback strategy for handling client certificates and private keys via secure_cert_key_paths in _mtls_helper.py. Feedback on these changes includes addressing a potential NameError in urllib3.py due to an unimported module, utilizing contextlib.ExitStack instead of manual context manager calls to prevent resource leaks, and verifying write permissions for /dev/shm to avoid runtime crashes in restricted environments.

gemini-code-assist · 2026-06-17T18:01:30Z

+    if sys.platform == "linux" and hasattr(os, "memfd_create"):
+        cm = _memfd_cert_key_paths(cert_bytes, key_bytes)
+        try:
+            cert_path, key_path = cm.__enter__()
+        except OSError:
+            pass  # Fallback to Tier 3 on failure.
+        else:
+            try:
+                # Handle cases where path exists but might be restricted.
+                if (cert_path is None or os.path.exists(cert_path)) and (
+                    key_path is None or os.path.exists(key_path)
+                ):
+                    yield cast(str, cert_path or cert), cast(
+                        str, key_path or key
+                    ), passphrase
+                    return
+            finally:
+                import sys
+
+                exc_info = sys.exc_info()
+                cm.__exit__(
+                    *(exc_info if exc_info[0] is not None else (None, None, None))
+                )
+            # If verification failed, fall through to Tier 3.


Manual context manager entry and exit (__enter__ and __exit__) is error-prone and can lead to resource leaks or incorrect exception propagation. Using contextlib.ExitStack is a much safer and more idiomatic way to handle conditional context managers in Python.

Suggested change

if sys.platform == "linux" and hasattr(os, "memfd_create"):

cm = _memfd_cert_key_paths(cert_bytes, key_bytes)

try:

cert_path, key_path = cm.__enter__()

except OSError:

pass # Fallback to Tier 3 on failure.

else:

try:

# Handle cases where path exists but might be restricted.

if (cert_path is None or os.path.exists(cert_path)) and (

key_path is None or os.path.exists(key_path)

):

yield cast(str, cert_path or cert), cast(

str, key_path or key

), passphrase

return

finally:

import sys

exc_info = sys.exc_info()

cm.__exit__(

*(exc_info if exc_info[0] is not None else (None, None, None))

)

# If verification failed, fall through to Tier 3.

if sys.platform == "linux" and hasattr(os, "memfd_create"):

memfd_failed = False

with contextlib.ExitStack() as stack:

try:

cert_path, key_path = stack.enter_context(

_memfd_cert_key_paths(cert_bytes, key_bytes)

)

except OSError:

memfd_failed = True

if not memfd_failed:

# Handle cases where path exists but might be restricted.

if (cert_path is None or os.path.exists(cert_path)) and (

key_path is None or os.path.exists(key_path)

):

yield cast(str, cert_path or cert), cast(

str, key_path or key

), passphrase

return

gemini-code-assist · 2026-06-17T18:01:30Z

+    with _mtls_helper.secure_cert_key_paths(cert, key) as (
+        cert_path,
+        key_path,
+        passphrase,
+    ):


The module _mtls_helper is not imported directly in this file; instead, it is accessed via transport._mtls_helper (as seen on line 353). Using _mtls_helper directly here will result in a NameError at runtime when _make_mutual_tls_http is called.

Suggested change

with _mtls_helper.secure_cert_key_paths(cert, key) as (

cert_path,

key_path,

passphrase,

):

with transport._mtls_helper.secure_cert_key_paths(cert, key) as (

cert_path,

key_path,

passphrase,

):

gemini-code-assist · 2026-06-17T18:01:30Z

+    import tempfile
+
+    # Prioritize RAM-backed /dev/shm to avoid writing secrets to physical storage.
+    tmp_dir = "/dev/shm" if os.path.isdir("/dev/shm") else None


In restricted or containerized environments (such as certain Docker/Kubernetes setups or serverless environments like Cloud Run), /dev/shm might exist but not be writable by the current user. If tempfile.mkstemp is called with a non-writable directory, it will raise a PermissionError and crash. Checking os.access for write permissions ensures a safe fallback to the default temp directory.

Suggested change

tmp_dir = "/dev/shm" if os.path.isdir("/dev/shm") else None

tmp_dir = (

"/dev/shm"

if os.path.isdir("/dev/shm") and os.access("/dev/shm", os.W_OK)

else None

)

daniel-sanche · 2026-06-17T23:49:10Z

       bool: indicating whether the client certificate should be used for mTLS.
    """
    return _mtls_helper.check_use_client_cert()
+


Do all of these new methods need to be public?

macastelaz

The last commit, bc6d2b8, LGTM. I do also wonder if everything needs to be public or not though.

I am withholding approval on the PR as it currently contains much more than just this last commit.

lsirac

will post on line numbers

lsirac · 2026-07-01T20:53:14Z

+
+
+def load_client_cert_into_context(
+    ctx: ssl.SSLContext,


Statically annotating parameter ctx strictly as ssl.SSLContext contradicts the runtime duck-typing check at line 175 (hasattr(ctx, "load_cert_chain")). To align static typing with runtime duck typing, annotate ctx using typing.Protocol or Union[ssl.SSLContext, Any].

lsirac · 2026-07-01T20:53:14Z

+            passphrase_val,
+        ):
+            ctx.load_cert_chain(
+                certfile=cert_path, keyfile=key_path, password=passphrase_val


Calling ctx.load_cert_chain with password=passphrase_val allows None to be passed, whereas aio/transport/mtls.py:61, requests.py:217, and urllib3.py:182 convert None to an empty string "" (password=passphrase_val or "") to prevent TypeError across different OpenSSL C bindings.

lsirac · 2026-07-01T20:53:14Z

+            ctx.load_cert_chain(
+                certfile=cert_path, keyfile=key_path, password=passphrase_val
+            )
+    except (


The exception handler for ctx.load_cert_chain() omits TypeError. If cert_bytes, key_bytes, or passphrase is None (or an invalid type), Python's ssl module raises TypeError, which escapes uncaught instead of raising the documented MutualTLSChannelError.

lsirac · 2026-07-01T20:53:14Z

+        ValueError,
+        RuntimeError,
+    ) as caught_exc:
+        new_exc = exceptions.MutualTLSChannelError(


Raising MutualTLSChannelError with a static string ("Failed to load client certificate and key for mTLS.") instead of passing the underlying caught_exc is inconsistent with lines 75 and 120, losing low-level OpenSSL failure specifics when exception messages are logged without tracebacks.

lsirac · 2026-07-01T20:53:14Z

+        cert_bytes,
+        key_bytes,
+        passphrase,
+    ) = _mtls_helper.get_client_ssl_credentials()


Invoking _mtls_helper.get_client_ssl_credentials() in load_default_client_cert without exception handling allows raw OSError, RuntimeError, and ValueError exceptions to escape out of get_default_ssl_context() (line 277), violating the function's documented contract.

lsirac · 2026-07-01T20:53:14Z

+        ("invalid_val", False, False),
+    ],
+)
+@mock.patch(


In tests evaluating environment variable lookups, the code patches the module string constant GOOGLE_API_USE_MTLS_ENDPOINT with its own literal string value ("GOOGLE_API_USE_MTLS_ENDPOINT"), which is redundant boilerplate.

lsirac · 2026-07-01T20:53:14Z

+    @mock.patch.object(os, "memfd_create", create=True)
+    @mock.patch.object(_mtls_helper, "_memfd_cert_key_paths", autospec=True)
+    @mock.patch.object(_mtls_helper, "_tempfile_cert_key_paths", autospec=True)
+    def test_tier2_fallback_to_tier3_on_oserror(


While test_tier2_fallback_to_tier3_on_oserror verifies fallback when RAM descriptors fail, the test suite lacks a negative test case verifying exception propagation when Tier 3 disk fallback (_tempfile_cert_key_paths) fails with OSError or ValueError.

lsirac · 2026-07-01T20:53:14Z

+        mock_close.assert_called_once_with(15)
+
+
+class TestTempfileCertKeyPaths(object):


In TestTempfileCertKeyPaths, no test verifies that an exception raised during key_path creation or writing triggers cleanup of the already-created cert_path file in the context manager's error handler.

lsirac · 2026-07-01T20:53:14Z

+        decrypted = serialization.load_pem_private_key(
+            encrypted_bytes, password=b"my_passphrase"
+        )
+        assert decrypted


In test_encrypt_key_if_plaintext, assert decrypted performs a weak truthy check on the loaded private key object rather than serializing and comparing key parameters against pytest.private_key_bytes to verify exact key equality.

lsirac · 2026-07-01T20:53:14Z

+    @mock.patch("builtins.open", autospec=True)
+    @mock.patch.object(os, "fsync")
+    @mock.patch.object(os, "remove")
+    def test_success(


In TestSecureWipeAndRemove, the test verifies write(), fsync(), and remove(), but never asserts mock_fh.flush.assert_called_once(), even though _secure_wipe_and_remove explicitly calls f.flush() before os.fsync().

nbayati and others added 10 commits June 10, 2026 06:07

fix: resolve mypy and lint issues

48f4e26

fix: suppress interactive OpenSSL stdin passphrase prompts during mTL…

41282b8

…S cert loading fallbacks

add unit tests to mtls_helper

1704dd8

refactor(auth): use os.fdopen for writing to memfd in mtls helper and…

a6882e4

… update tests accordingly

test(auth): fix failing test by updating mock_mds_mtls_config asserti…

f9a741a

…on to include empty password for client_combined_cert_path

fix nox failures

04776d6

test: add edge case and error handling tests for _mtls_helper functions

980527e

fix lint error, again!

105b02a

nbayati requested review from a team as code owners June 17, 2026 17:59

gemini-code-assist Bot reviewed Jun 17, 2026

View reviewed changes

daniel-sanche reviewed Jun 17, 2026

View reviewed changes

daniel-sanche changed the title ~~Implement python mtls helpers~~ feat(auth): Implement python mtls helpers Jun 17, 2026

macastelaz reviewed Jun 25, 2026

View reviewed changes

lsirac reviewed Jul 1, 2026

View reviewed changes

lsirac requested changes Jul 1, 2026

View reviewed changes

lsirac approved these changes Jul 1, 2026

View reviewed changes

		mock_close.assert_called_once_with(15)


		class TestTempfileCertKeyPaths(object):

Uh oh!

Conversation

nbayati commented Jun 17, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macastelaz left a comment

Choose a reason for hiding this comment

Uh oh!

lsirac left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lsirac left a comment •

edited

Loading