Apiserver heap corruption under high-concurrency LIST load (~7–9k req/s) — pre-existing, arch-independent

[scotwells]

Summary

Under sustained high-concurrency LIST load (~7–9k req/s, list-heavy), the ipam-apiserver suffers process-wide Go heap corruption and crashes (CrashLoopBackOff), returning a 503 storm through the aggregation layer. This is a real, architecture-independent, pre-existing stability bug in the shared serving path — not introduced by the IPClass work (PR #70), which is proven race-free. Filed as a decoupled follow-up so the validated IPClass feature can land; the existing MaxConns=10 mitigation in cmd/ipam/serve.go keeps normal-load exposure low.

Severity

High — apiserver process crash under load; affects availability of all IPAM resources during read spikes.

Evidence it is real (not a test artifact)

• Reproduces on two architectures: amd64 (emulated) ~9,300 req/s → ipam_read_success_rate 0.29%, restarts; native arm64 (no emulation) ~7,026 req/s → 1.67% success, both pods CrashLoopBackOff. Rules out qemu emulation.
• Crash signature: fatal error: found bad pointer in Go heap / found pointer to free object + SIGSEGV on fault addresses that are ASCII string bytes used as pointers (e.g. 0x64726f63657280="record", "finish ", "…prefix"). Classic single-unsynchronized-writer-over-shared-heap data race.
• Process-wide corruption: the fatal fault lands in a different subsystem each crash — JSON encode of the discovery doc, http.Header.Clone on an outbound client-go auth roundtrip, LIST decode metav1.Time.UnmarshalJSON, TLS handshake (cryptobyte), APF queueSet — i.e. the crashing goroutine is a victim, not the cause.
• Disk ruled out: reproduced with /var bounded (audit-log rotation cap applied); not OOM (mem limit 4Gi, ample).

Not IPAM code

• Zero IPAM frames in any crashing goroutine stack (grep miloapis in stacks = 0).
• IPAM read/store/convert/watch paths proven race-free by exhaustive go test -race (added in PR docs: propose IPClass, a policy layer for claiming IP space #70): concurrent Store.GetList decode→convert→encode (JSON + protobuf) against real Postgres, live watchers + readers + writers, and the apiserver codec — all clean.
• The IPClass conversion diff is purely additive read-only copies; it only enlarges IPPool objects slightly (more GC/alloc pressure), an aggravator at most.

Pre-existing

cmd/ipam/serve.go on origin/main already documents this exact bug: MaxConns is pinned to 10 as a mitigation for "intermittent heap corruption seen under sustained ~4–8k req/s load … inside Go's stdlib context.(*cancelCtx).propagateCancel map assignment … concurrency-induced runtime state corruption that surfaces only when many request goroutines overlap." The read spike at ~9k req/s exceeds that ceiling and re-exposes it.

Why the race detector can't name it (heisenbug)

• Offline go test -race and a live -race build both came back clean. The -race binary is ~25× slower (~395 req/s), far below the crash threshold, so it never reaches the interleaving. No positive control available via -race.

Prime suspect: OTel HTTP instrumentation

• Every crashing run has the apiserver's OTel instrumentation active (inbound otelhttp WithTracing middleware + the outbound delegated-auth otelhttp.Transport wrapping — the Header.Clone frame). This is driven by the APIServerTracing feature gate, which is LockToDefault:true on k8s 1.35 (vendored k8s.io/apiserver v0.35.3) — so it cannot be disabled by config (serve doesn't even expose --feature-gates).
• Deps: go.opentelemetry.io/otel v1.43.0, otelhttp v0.68.0, otelpgx v0.11.1. otelhttp had data-race fixes but in the v0.48–v0.54 era (already included in v0.68.0); no obvious post-v0.68 fix found — a dependency bump is not a confirmed fix.

Reproduction

1. Deploy IPAM to kind.
2. task test/load:setup
3. task test/load:reads (0→100 VU list-heavy spike).
   Crash is deterministic within one run once concurrency reaches ~7k+ req/s.

Suggested next steps

• Confirm the OTel-instrumentation hypothesis with a code-patched build that strips the otelhttp wrapping (inbound WithTracing + outbound auth transport); this lives in vendored k8s.io/apiserver + client-go, so it's a non-trivial seam.
• If confirmed: evaluate (a) a targeted apiserver-wiring change to avoid the crashing otel path (cost: lose apiserver tracing), (b) an otel dependency bump (verify newest changelog), or (c) upstream escalation to k8s/otel.
• Keep the MaxConns=10 mitigation until fixed.

Relationship to PR #70 (IPClass)

Decoupled. PR #70 is feature-complete and race-clean (e2e green; 3/4 perf thresholds pass — throughput 147/s, class-claim 93.9/s, exhaustion-deny; only read-success-rate blocked by this bug). Race-regression tests guarding the store/convert path shipped with PR #70.

[scotwells]

Fix-feasibility assessment (from the diagnosis team)

No confirmed small in-repo fix today, but a cheap first lever is worth trying before any heavy path:

1. Cheapest — bump delegated-auth webhook cache TTLs (one deployment flag, no code). serve registers --authentication-token-webhook-cache-ttl, --authorization-webhook-cache-authorized-ttl, --authorization-webhook-cache-unauthorized-ttl (default 10s). Cranking them to ~1h nearly eliminates the outbound TokenReview/SAR roundtrips — i.e. the outbound otelhttp.Transport auth path that appears in a fatal crash frame. This is both a discriminator and a potential landable mitigation: if the crash stops, the outbound auth path is confirmed the writer and the TTL bump is the fix; if it persists, the outbound path is cleared and the racer is the inbound WithTracing middleware or elsewhere. Caveat: only helps if the writer is the outbound path (inbound middleware runs on every request regardless).

2. Narrow in-repo seam (only if outbound is the culprit). DelegatingAuthentication/AuthorizationOptions.WithCustomRoundTripper(transport.WrapperFunc) may let us install a plain (non-otelhttp) transport for the delegated-auth clients, removing the crashing outbound wrapping from IPAM's own wiring — no fork. Unverified whether it composes-over vs replaces the otel wrapping. Cost if it works: lose tracing on auth egress only (minor).

3. Heavy paths (if 1 & 2 fail). Inbound WithTracing is inserted by k8s.io/apiserver DefaultBuildHandlerChain (config.go:1051), gated only by the GA-locked APIServerTracing gate — no flag/config lever, and IPAM's BuildHandlerChainFunc only wraps the default chain (can't remove that filter). Removing it means reimplementing the default chain minus tracing or a go mod replace fork of k8s.io/apiserver — carry-a-fork maintenance, not a small diff. Dep bump: no known otel/otelhttp race fix after v0.68 (searched); low confidence.

Suggested order: try (1) → if it confirms outbound, try (2) for a clean in-repo fix → else (3)/upstream. Keep MaxConns=10 meanwhile.

A high-VU (1000–2000) -race run was also queued to try to trip tsan and name the writer for free (the 100-VU -race run was too slow at ~395 req/s to reach the ~9k crash threshold).

[scotwells]

Update — high-VU -race path exhausted. The queued diagnostic (re-run the -race image at 1000–2000 VUs to try to trip tsan) was run: a genuine 1500-VU spike against the race-instrumented binary was a solid negative — no DATA RACE warning, no crash. So the race detector cannot name this writer even at high concurrency (the heisenbug is hidden by tsan's serialization regardless of VU count, not merely by the -race binary's lower throughput). That confirmation path is exhausted; the remaining live options are the auth-cache-TTL discriminator (cheapest, doubles as a candidate fix) and the otelhttp-strip vendored build — see the fix-feasibility comment above.

[scotwells]

Refined hypothesis — likely NOT a tsan-visible data race. With -race now clean at both low and high concurrency (100 VUs and a sustained ~1500-VU/~8-min run, 0 restarts), combined with the crash signature (found pointer to free object, string-bytes-used-as-pointers), the leading hypothesis shifts away from a classic Go data race that the race detector would name, toward something tsan does not instrument: unsafe.Pointer misuse or a sync.Pool'd / vendored fast-path buffer reused-after-free on the hot serving path.

Revised next steps for #71:

• Don't expect -race to localize it — that path is exhausted.
• Audit pooled/unsafe buffers in the serving hot path (serializer/framer buffer pools, transport/tls buffers, the otelhttp/client-go egress path) for reuse-after-free under overlap.
• Use the auth-cache-TTL flag as the cheap outbound-path discriminator (crash stops ⇒ the outbound otelhttp auth path is implicated).

The MaxConns=10 mitigation on main remains the interim guard.