Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Collaborator

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-client ConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
  endpoints: ["https://etcd.aisix.cloud:2379"]
  tls:
    ca_cert_file:     "/etc/aisix/mtls/ca.crt"
    client_cert_file: "/etc/aisix/mtls/client.crt"
    client_key_file:  "/etc/aisix/mtls/client.key"
    # domain_name: "etcd.aisix.cloud"  # optional; defaults to endpoints[0] host
proxy:
  addr: "0.0.0.0:3000"
managed:
  enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

Component Standalone Managed
Proxy listener ✅ bound ✅ bound
etcd provider + watch supervisor
Admin etcd client (write path) ❌ skipped
Admin router (/admin/v1/*, UI, Playground) ✅ bound ❌ never constructed
Admin TCP listener ✅ bound ❌ never bound
/metrics (currently mounted on admin) ✅ served ❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)

Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.

## What's new

### `aisix-core::config`

- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
  Three PEM file paths (CA cert, client cert, client key) plus an
  optional `domain_name` for SNI. Defaults derive the domain from the
  first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
  switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
  can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
  invariants when `managed.enabled = true`, and keeps them as-is
  otherwise — no silent regression for standalone setups.

### `aisix-server::main`

- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
  helper. Returns `None` for plain HTTP (keep the test path cheap),
  wires `with_user` + `with_tls` when present, surfaces missing
  cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
  endpoint strings (`http://host:port`, `https://host:port`, bare
  `host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
  now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
  surface is never built:
    * `admin_client` stays `None` (no second etcd connection)
    * `admin_state` / `admin_router` are not constructed
    * The admin TCP listener is not bound
    * The Playground endpoint (mounted inside admin) vanishes
  The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
  managed-mode start-up no longer joins on a nonexistent future.

### `config.example.yaml`

- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
  explanation of what flips in that mode.

## Tests

### `aisix-core`

- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
  tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
  original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.

### `aisix-server`

- `default_domain_strips_scheme_port_and_brackets`: table for the
  SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
  doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
  sees *which* file is missing without grepping filesystem state.

`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).

## Explicitly out of scope (follow-up PRs)

- DP registration flow (`POST /dp/register` against cp-api) that
  *fetches* the mTLS bundle and persists it. This PR expects the
  bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
  connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
Copilot AI review requested due to automatic review settings April 23, 2026 07:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-client ConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
crates/aisix-server/src/main.rs Build etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rs Re-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rs Extend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yaml Document new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if both etcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match (etcd.user.as_ref(), etcd.password_env.as_ref()) {
(Some(user), Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None, None) => {}
(Some(_), None) => {
return Err(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None, Some(_)) => {
return Err(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
if let Some(dial_timeout_ms) = etcd.dial_timeout_ms {
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
if let Some(request_timeout_ms) = etcd.request_timeout_ms {
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicely contextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0 {
return Err(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).

Retention tightened per artifact by expected re-read horizon:

  - aisix-bin       1 day   (consumed by the same-day e2e job only)
  - ui-dist         7 days  (consumed by same-day e2e; light enough
                             to keep a week for manual inspection)
  - coverage-*      7 days  (manual download for debugging flaky
                             coverage gates; LCOV is tiny)

Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.

No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into main Apr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:

- tool_calls / Anthropic tool_use (normalized into message.extra) are
  now folded into a single guardrail-inspected text view via
  ChatResponse::guardrail_output_text(), used by the keyword, text-
  moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
  Reasoning/thinking content is intentionally left out of scope.

- Non-streaming cache hits now run the resolved output guardrail chain
  before returning the stored body, instead of replaying it unchecked
  (#28). Streaming output guardrails already run end-of-stream.

Part of #448 (findings #3, #18, #21, #28)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants