feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28
Conversation
…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
There was a problem hiding this comment.
Pull request overview
Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.
Changes:
- Add
etcd.tlsconfig (CA/cert/key + optionaldomain_name) and buildetcd-clientConnectOptionswith mTLS. - Add
managed.enabledconfig and skip admin etcd client + admin router/listener when managed mode is on. - Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| crates/aisix-server/src/main.rs | Build etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode. |
| crates/aisix-core/src/lib.rs | Re-export new config types (EtcdTlsConfig, ManagedConfig). |
| crates/aisix-core/src/config.rs | Extend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules. |
| config.example.yaml | Document new etcd.tls block and managed.enabled toggle. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) { | ||
| let pw = std::env::var(env_key).map_err(|_| { | ||
| anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing") | ||
| })?; | ||
| options = options.with_user(user.clone(), pw); | ||
| needs_options = true; |
There was a problem hiding this comment.
build_etcd_connect_options() only applies basic auth if both etcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.
| if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) { | |
| let pw = std::env::var(env_key).map_err(|_| { | |
| anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing") | |
| })?; | |
| options = options.with_user(user.clone(), pw); | |
| needs_options = true; | |
| match (etcd.user.as_ref(), etcd.password_env.as_ref()) { | |
| (Some(user), Some(env_key)) => { | |
| let pw = std::env::var(env_key).map_err(|_| { | |
| anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing") | |
| })?; | |
| options = options.with_user(user.clone(), pw); | |
| needs_options = true; | |
| } | |
| (None, None) => {} | |
| (Some(_), None) => { | |
| return Err(anyhow::anyhow!( | |
| "etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth" | |
| )); | |
| } | |
| (None, Some(_)) => { | |
| return Err(anyhow::anyhow!( | |
| "etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth" | |
| )); | |
| } |
| fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> { | ||
| let mut needs_options = false; | ||
| let mut options = ConnectOptions::new(); | ||
|
|
There was a problem hiding this comment.
EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.
| if let Some(dial_timeout_ms) = etcd.dial_timeout_ms { | |
| options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms)); | |
| needs_options = true; | |
| } | |
| if let Some(request_timeout_ms) = etcd.request_timeout_ms { | |
| options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms)); | |
| needs_options = true; | |
| } |
| /// failure bubbles up as a nicely-contextualised BootstrapError at | ||
| /// the same point as other etcd connection errors. |
There was a problem hiding this comment.
The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.
| /// failure bubbles up as a nicely-contextualised BootstrapError at | |
| /// the same point as other etcd connection errors. | |
| /// failure bubbles up as a nicely contextualised error at the same | |
| /// point as other etcd connection errors. |
| if self.admin.addr.parse::<std::net::SocketAddr>().is_err() { | ||
| return Err(BootstrapError::Config(format!( | ||
| "admin.addr invalid socket address: {}", | ||
| self.admin.addr | ||
| ))); |
There was a problem hiding this comment.
AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.
| if self.admin.addr.parse::<std::net::SocketAddr>().is_err() { | |
| return Err(BootstrapError::Config(format!( | |
| "admin.addr invalid socket address: {}", | |
| self.admin.addr | |
| ))); | |
| let admin_addr = self | |
| .admin | |
| .addr | |
| .parse::<std::net::SocketAddr>() | |
| .map_err(|_| { | |
| BootstrapError::Config(format!( | |
| "admin.addr invalid socket address: {}", | |
| self.admin.addr | |
| )) | |
| })?; | |
| if admin_addr.port() == 0 { | |
| return Err(BootstrapError::Config( | |
| "admin.addr must use an explicit non-zero port \ | |
| (required when managed.enabled is false)" | |
| .into(), | |
| )); |
Free-tier Actions storage is 500 MB, shared across the whole repo. Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run, so storage saturates after <10 main-branch pushes and blocks \`actions/upload-artifact\` on every subsequent PR (the failure that paused #28 twice today). Retention tightened per artifact by expected re-read horizon: - aisix-bin 1 day (consumed by the same-day e2e job only) - ui-dist 7 days (consumed by same-day e2e; light enough to keep a week for manual inspection) - coverage-* 7 days (manual download for debugging flaky coverage gates; LCOV is tiny) Nothing downstream relies on week+ old binaries — \`build-bin\` is re-runnable from source and the \`needs:\` chain on \`build-aisix (instrumented)\` → \`e2e\` already re-produces the artifact when an earlier run has expired. No behaviour change on passing runs; only limits how long stale runs squat on quota.
Output guardrails only inspected message.content, so client-visible output that lives elsewhere bypassed content/DLP checks: - tool_calls / Anthropic tool_use (normalized into message.extra) are now folded into a single guardrail-inspected text view via ChatResponse::guardrail_output_text(), used by the keyword, text- moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21). Reasoning/thinking content is intentionally left out of scope. - Non-streaming cache hits now run the resolved output guardrail chain before returning the stored body, instead of replaying it unchecked (#28). Streaming output guardrails already run end-of-stream. Part of #448 (findings #3, #18, #21, #28)
Summary
Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:
etcd-clientConnectOptions.This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.
Config shape
Standalone users are unaffected: new fields are all optional and defaulted,
validate()keeps the existing invariants whenmanaged.enabled = false.What the managed switch actually flips
/admin/v1/*, UI, Playground)/metrics(currently mounted on admin)The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.
New helper
build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:Nonefor plain HTTP etcd (keeps the hot path cheap — no pointlessConnectOptions::new()allocation when nothing needs wiring).TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.default_domain_from_endpoint()extracts the SNI from URL-ish strings:http://,https://, barehost:port, and IPv6 literals[::1]:2379— table-tested.Tests
cargo test --workspace --all-featuresgreen (407 tests). New cases:aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted— minimum aisix.cloud tenant YAML loads cleanly.aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false— original invariant preserved.aisix-core::config::tests::parses_etcd_tls_block— round-trip for all four TLS fields.aisix-server::tests::default_domain_strips_scheme_port_and_brackets— SNI extractor table.aisix-server::tests::build_connect_options_none_when_plain_http— hot path stays zero-cost.aisix-server::tests::build_connect_options_surfaces_missing_cert_files— operator-friendly error.cargo fmt+cargo clippy --workspace --all-targets -- -D warningsclean.Explicitly out of scope (follow-up PRs on this branch's successor stack)
POST /dp/registerclient that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.POST /api/ai_dataplane/heartbeatso cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).Relationship
Paired with the aisix.cloud side in
api7/AISIX-Cloud#8: that PR'sGatewayHandlers.createcallsIssueDataplaneCertificateon api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.