Skip to content

[Hashicorp] Fix Vault GCP auth for GCE metadata credentials#68069

Merged
potiuk merged 1 commit into
apache:mainfrom
fpopic:fix_vault_gcp_auth_requirements
Jun 11, 2026
Merged

[Hashicorp] Fix Vault GCP auth for GCE metadata credentials#68069
potiuk merged 1 commit into
apache:mainfrom
fpopic:fix_vault_gcp_auth_requirements

Conversation

@fpopic

@fpopic fpopic commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What

Fix HashiCorp Vault GCP authentication when Application Default Credentials come from Compute Engine metadata credentials and initially expose the service account email as default.

Why

In GCE-based environments such as Cloud Composer, google.auth.compute_engine.Credentials may start with service_account_email == "default" until the credentials are refreshed from the metadata server. The Vault client used that value directly in the IAM signJwt request, producing an invalid resource name like projects/<project>/serviceAccounts/default.

Sanitized task log excerpt:

{connection.py:531} ERROR - Unable to retrieve connection from secrets backend (VaultBackend). Checking subsequent secrets backend.
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/<project-id>/serviceAccounts/default:signJwt?alt=json returned "Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account". Details: "Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account">
airflow.exceptions.AirflowNotFoundException: The conn_id `<conn-id>` isn't defined

Context

How

  • Resolve GCP service account email before building the IAM signJwt request.
  • Treat missing or default service account email values as unresolved.
  • Refresh Compute Engine credentials so the metadata server populates the real service account email.
  • Preserve key-file behavior by using client_email when available.
  • Add a regression test for the Compute Engine/Composer ADC case.

Tests

.venv/bin/python -m pytest providers/hashicorp/tests/unit/hashicorp/_internal_client/test_vault_client.py -q --with-db-init

AI assistance

This PR was prepared with help from GPT-5 / Codex.

@fpopic fpopic requested a review from hussein-awala as a code owner June 5, 2026 09:14
@boring-cyborg boring-cyborg Bot added area:providers provider:hashicorp Hashicorp provider related issues labels Jun 5, 2026
@fpopic fpopic force-pushed the fix_vault_gcp_auth_requirements branch from fddeb58 to ac64fd6 Compare June 5, 2026 09:25
@fpopic fpopic marked this pull request as draft June 5, 2026 09:39
@fpopic fpopic force-pushed the fix_vault_gcp_auth_requirements branch from ac64fd6 to 8513abb Compare June 5, 2026 09:47
@fpopic fpopic marked this pull request as ready for review June 5, 2026 09:54
@fpopic

fpopic commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

@potiuk it is ready for review, once released as rc2 will do a real intergration test in our enterprise environment.

@fpopic fpopic changed the title [Hashicorp] Fix Vault GCP auth for Compute Engine metadata credentials [Hashicorp] Fix Vault GCP auth for GCE metadata credentials Jun 5, 2026
@potiuk potiuk merged commit d31f665 into apache:main Jun 11, 2026
225 of 230 checks passed
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Jun 16, 2026
dingo4dev pushed a commit to dingo4dev/airflow that referenced this pull request Jun 16, 2026
@fpopic fpopic deleted the fix_vault_gcp_auth_requirements branch June 16, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:hashicorp Hashicorp provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants