Add proxy support to Databricks connections

[Vamsi-klu]

Add explicit proxy support for Databricks connections.

What changed

This adds a documented proxies connection extra for the Databricks provider. The extra is intentionally allowlisted and only accepts http and https keys, for example:

{
  "proxies": {
    "http": "http://proxy.example.com:8080",
    "https": "http://proxy.example.com:8443"
  }
}

The configured proxy is applied to:

• Databricks REST API calls
• Databricks OAuth token exchanges
• Azure Identity token acquisition for AAD service principal and default Azure credential authentication
• Sync and async Databricks hook paths

Why

Users running Airflow behind a corporate proxy could not authenticate to Azure Databricks because token acquisition to Azure/Databricks endpoints did not receive proxy configuration.

This fixes #32576 while keeping connection extras explicit and safe. The implementation does not pass arbitrary connection extras through to client libraries; it validates and forwards only the reviewed proxies extra.

Testing

I added coverage for:

• Valid proxies parsing from Databricks connection extras
• Rejection of invalid proxy shapes and unsupported keys
• Sync Databricks REST API calls using proxies
• Async Databricks REST API calls using the matching aiohttp proxy
• AAD service principal token paths passing proxies to Azure Identity
• Databricks service-principal OAuth token exchange using proxies
• Sync and async token/API combinations
• Databricks SQL connector kwargs not receiving proxies

Commands run:

• uv run ruff format providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py
• uv run ruff check --fix providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py
• uv run ruff format providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
• uv run ruff check --fix providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
• uv run ./scripts/ci/prek/check_new_airflow_exception_usage.py providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py
• uv run --project providers/databricks pytest providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py -q

Result: 284 passed; AirflowException usage check passed.

closes: #32576

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

---

Drafted-by: Codex (GPT-5) (no human review before posting)

[eladkal]

cc @moomindani @mwojtyczka for review from Databricks team

[Vamsi-klu]

Databricks proxy support validation for PR #68527

Validated against exact PR head 124d736716ec26bc58bdcd71d57335b6b5998893 on June 21, 2026.

Test suite

Focused Databricks provider tests passed:

284 passed, 1 warning in 7.85s

Command:

PYTHONPATH=/Users/nalam/Downloads/airflow-68527-vr/airflow-core/src:/Users/nalam/Downloads/airflow-68527-vr/providers/databricks/src \
  /Users/nalam/airflow-1/.venv/bin/python -m pytest \
  providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py \
  providers/databricks/tests/unit/databricks/hooks/test_databricks.py \
  providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py -q

Live Databricks validation

Created a one-day personal access token scoped to clusters, used it only for this validation, and revoked it after the run. The access token page returned to No tokens exist. after cleanup.

The live validation used Airflow's DatabricksHook from the PR worktree with connection extra:

{
  "proxies": {
    "http": "http://127.0.0.1:<port>",
    "https": "http://127.0.0.1:<port>"
  }
}

The validation called GET /api/2.0/clusters/list against the Databricks workspace through a local HTTP CONNECT proxy.

Sync path result:

sync_api_result.ok=true
sync_api_result.cluster_count=0
sync_proxy_tunnel_to_workspace_observed=true
CONNECT dbc-c1a3fd5e-8c66.cloud.databricks.com:443 HTTP/1.1

Async path result:

async_api_result.ok=true
async_api_result.cluster_count=0
async_proxy_tunnel_to_workspace_observed=true
CONNECT dbc-c1a3fd5e-8c66.cloud.databricks.com:443 HTTP/1.1

This covers the PR's sync requests forwarding path and async aiohttp forwarding path with an authenticated Databricks API call.

Evidence

Evidence bundle: https://gist.github.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22

Validation JSON: https://gist.githubusercontent.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22/raw/70dc93e752813464937281658b7edc34d2c942b1/databricks_proxy_live_validation.json

Repro script: https://gist.githubusercontent.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22/raw/70dc93e752813464937281658b7edc34d2c942b1/databricks_proxy_live_validation.py

Workspace availability:

Workspace page:

Post-cleanup token state:

[jroachgolf84]

LGTM.

[Vamsi-klu]

Thanks @jroachgolf84 for the approval, @eladkal Can i get a stamp please? Thanks for the time!

[moomindani]

Checked the following:

• The proxies extra is allowlisted (http/https only) with validation — arbitrary connection extras are not passed through.
• Proxy is applied to sync/async REST API calls, Azure AAD credential acquisition (ClientSecretCredential, DefaultAzureCredential), and Databricks OAuth token exchange. Intentionally excluded from SQL connector kwargs and ManagedIdentityCredential (IMDS is a local metadata service).
• databricks-sdk and databricks-sql-connector do not natively support HTTP proxy configuration, so this connection-extra approach fills a real gap.

One question and one minor item:

• Could the docs clarify the relationship with HTTP_PROXY/HTTPS_PROXY environment variables? requests and aiohttp already honor those, so it would help users understand when the explicit proxies extra is needed — specifically, which paths (e.g. Azure AAD token acquisition via Azure Identity SDK) don't pick up the environment variables and require the explicit setting, and that this extra overrides the environment variables when both are set.
• Consider adding a brief comment on why ManagedIdentityCredential is excluded from proxy kwargs — the other Azure credential paths receive them, so the asymmetry might confuse a future reader.

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @moomindani before posting

[Vamsi-klu]

Thanks @moomindani for the careful review — both points addressed in the latest commit (58b7c4a).

1. Docs: added a Note under the proxies extra explaining the relationship with HTTP_PROXY / HTTPS_PROXY / NO_PROXY. One correction worth surfacing from tracing the code: the paths that do not read the proxy env vars are the async aiohttp paths, not the Azure Identity ones. The hook builds aiohttp.ClientSession() without trust_env=True, so the async REST and token-exchange paths ignore HTTP_PROXY/HTTPS_PROXY entirely — the proxies extra is the only way to proxy them. The sync requests paths and the Azure Identity SDK paths (azure-core's transport defaults to use_env_settings=True) both honor the env vars by default, and the proxies extra takes precedence over them when both are set. The Note documents this per client.

2. Comment: added a comment at both ManagedIdentityCredential call sites (sync _get_aad_token and async _a_get_aad_token) explaining the exclusion — managed identity authenticates against the link-local IMDS endpoint (169.254.169.254), which must be reached directly and is unsupported behind a proxy, so the proxies extra is intentionally not forwarded there, unlike the ClientSecretCredential / DefaultAzureCredential paths that reach the public Entra ID endpoint. (The IMDS metadata-service call is likewise left unproxied.)

Also added a regression test asserting ManagedIdentityCredential and the IMDS metadata call are constructed without proxies, while the Databricks REST call still receives it — so the asymmetry can't silently regress.

No changelog/newsfragment: providers don't consume newsfragments, and this is a docs + comment clarification with a test, no behavior change. Re-ran the databricks hook tests (285 passed).

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @Vamsi-klu before posting

[moomindani]

Thanks for the correction — you're right that the Azure Identity SDK paths honor the env vars; it's the async aiohttp paths that don't. The doc note, the ManagedIdentityCredential comments, and the regression test all look good to me.

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @moomindani before posting

[eladkal]

@moomindani any futher comments or are you approving this PR?

[moomindani]

No further comments. approved.