AE datavolume: DX-steered vs save-ALL reduction measurement

[entlein]

Stacked on #53 (ae-prod). Continues the AE datavolume work: now that the F1 10k-cap fix is live-verified on aeprod11 (passthrough firehose captures the full window, 42,516 http rows/query, read==wrote), this PR measures the datavolume reduction of DX-steered AE (rev-3 streaming — AE writes only the pods DX steers into its activeSet over the control surface) vs saving ALL data (passthrough firehose).

What

• New harness src/e2e_test/adaptive_export_loadtest/harness/exp_dx_steering_reduction.sh: two arms (ALL passthrough vs DX streaming), same fixed load, forensic_db active-part deltas (rows + on-disk bytes) per table; reduction = 1 - DX/ALL.
• Successor to ae_vs_all.sh (whose AE arm used the rev-2 controller gate + a stale malformed JNDI). This uses rev-3 DX streaming + the canonical resolvable JNDI FQDN so the chain fires and DX classifies.

Measurement (filled from the live run on the rig)

• ALL bytes/table, DX bytes/table, reduction % per table + total — to follow.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b8459eed-2eb0-4e79-866d-ca7de56eb6c3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ae-datavolume-dx-steering

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[entlein]

Run 1 — INVALID (methodology), not a reduction number

First DX-vs-ALL run on rig 6a32e7a9 (AE aeprod11, DX 0.2.0-rc5). The raw output showed "TOTAL reduction=100.0%" — but that is a dead DX arm artifact, not a real reduction. Root cause, three compounding issues:

1. Stale activeSet — the DX arm rehydrated adaptive_attribution export windows for pods that no longer exist (ld-12/22/25/28/31 = deleted F1-cap-test loaders, plus observer/postgres). AE's streaming scanner ran mode=whitelist pods=7 but every query returned rows=0 because it was querying dead pods.
2. log4shell chain did not fire — marshalsec Send LDAP reference count = 0, so the JNDI fire produced no LDAP egress → DX never got a log4shell signal → the backend (the traffic-bearing target) was never steered.
3. DX verdicts indeterminate (on observer), no log4shell-rce-exfil.

DX steering is mechanically alive (it classified + created export windows; AE activeSet populated, scanner in whitelist mode) — but the test let stale windows pollute the activeSet and the attack didn't fire.

Harness fixes before a valid number:

• Clear/expire adaptive_attribution at DX-arm start so the activeSet only contains freshly-steered, alive pods.
• Ensure the steered pod is alive and traffic-bearing for the whole DX window.
• For the attack-steered variant: confirm the log4shell chain actually fires (marshalsec serving + backend vulnerable + FQDN resolves) so DX issues a log4shell-rce-exfil verdict and steers the backend.

Next: fix the harness (clear stale windows + live-pod steering), re-run.

[entlein]

AE NFR benchmark — run 1 (aeprod11, rig 6a3383d0, passthrough firehose, 180s window)

Real metrics (forensic_db active-part deltas + kubectl top + ae_reconcile). Two metrics are excluded as measurement artifacts (fixes in progress, see notes) rather than reported wrong.

Throughput (AE write rate to forensic_db, under steady frontend load):

table	rows/s	bytes/s	rows/180s
http_events	33	2,120	5,893
conn_stats	21	315	3,864
dns_events	5	104	839

Resource footprint (sum of 2 AE pods):

state	CPU (m)	mem (Mi)
idle	429	27
under load	527	32

Write fidelity: write_err count = 0 across all passthrough cycles (http 7, conn 6, dns 7 cycles) → clean sink, no dropped batches. (read==wrote verified exactly in the dedicated F1 runs.)

Excluded (harness measurement bugs, not AE issues):

• Capture completeness % — the broker-count snapshot was taken after load ended vs AE cycles during the window (misaligned), yielding a nonsensical 623%. The cap/completeness proof is the dedicated F1 test (max_read > 10000, posted separately). Harness fix: align the broker window to the measurement window.
• E2E latency — now() - max(time_) hit a DateTime64 type error (na). Fix: dateDiff('second', max(time_), now()).

More NFR runs (variance) + the corrected completeness/latency to follow.

[entlein]

F1-cap re-confirm on fresh rig 6a3383d0 — read==wrote clean (conclusive >10k proof is on PR #53)

aeprod11 passthrough, 60 min window, ae_reconcile http cycles:

ts	read_count	wrote_count
06:26:54	7,181	7,181
06:26:33	5,926	5,926
06:25:56	4,854	4,854

read==wrote exactly, errs=0 — clean sink. Note this rig's broker in-memory retention (~6.5 min of http at the current rate) caps a single-window pull at ~7k, so I can't reproduce a >10,000 single-window read here. The conclusive cap-lift proof is on PR #53 (same aeprod11 binary, read=42,516 >> the old hard 10,000 cap). This re-confirm shows the binary behaves identically (uncapped up to what the broker holds, read==wrote) on a second rig.

[entlein]

DX-vs-ALL reduction — run 2 INVALID (rig node failure), not a number

Re-ran on 6a3383d0 with the attacker up + chain firing (marshalsec 5/5 pre-run). The run is invalid — node-01 went NotReady mid-run → TaintManagerEviction evicted the log4j-poc pods (backend vc2t2→lnmjg, observer, postgres, cl-* all Terminating/ContainerCreating). Consequences:

• backend restarted mid-run → JNDI hit a restarting pod → marshalsec_fires=0 at the guard.
• the two arms ran in different time windows under different (churning) load, so the deltas aren't comparable.
• byte-delta "reduction" came out negative (−226% http) — system.parts byte delta is compaction-noisy (e.g. pgsql −930), unsuitable as the primary metric.

What DID work (mechanism is sound): DX classified + steered the live backend into the activeSet (adaptive_attribution: backend = 3 windows) and AE streamed it (35,068 http rows in the DX arm). So DX→AE steering is functioning; the rig node failure invalidated the measurement.

Fixes for the valid run:

1. Cluster must be stable (node-01 Ready) — flagged to the rig owner.
2. Switch the reduction metric to rows (compaction-noise-free), measured over the SAME wall-clock window for both arms where possible.
3. Methodology: steering the backend (the dominant-traffic pod) bounds the reduction — will also report the backend's share of total cluster traffic so the number is interpretable.

Re-run pending node recovery.

[entlein]

DX-vs-ALL reduction — run 3 INVALID, root cause = upstream detection gap (bob#140), NOT AE

This time the cluster was stable (no node churn, all log4j-poc pods Running) and the chain fired 38× (marshalsec). Yet the DX arm wrote 0 rows → false "100%". Root cause is upstream of AE:

• DX steered observer (4 windows) + postgres (2) — NOT the backend (the log4shell target).
• dx-daemon produced verdicts only for observer/postgres (all triage=indeterminate); no backend verdict at all.
• → DX never gets a backend case, because kubescape doesn't flag the backend's log4shell (the R0011 "Unexpected Egress" DNS-gate suppresses DNS-named egress — filed bob#140; R0005 nuances). No kubescape anomaly on the backend ⇒ no DX referral ⇒ no steer ⇒ AE streams the wrong (low-traffic) pods ⇒ 0.

What this confirms: the DX→AE steering mechanism + AE streaming + the chain firing all work; the DX-steered-log4shell reduction is blocked on the upstream kubescape detection of log4shell (bob#140) — not on AE or DX. Reported, not faked.

To still get a real reduction number (independent of bob#140): directly steer the backend via the AE control surface and measure AE-streams-backend vs ALL (isolates the AE streaming-reduction mechanism). Pursuing that next.

(Resource hardening applied this cycle: AE measured 16–38Mi → capped 1Gi + cpu→1 core; PEM → 3Gi via pemMemoryLimit (was OOMing at the 2Gi default → PEM restarts disrupt AE). Node evictions were node-01 NotReady = network/heartbeat, not OOM.)

[entlein]

AE e2e evidence campaign — consolidated summary (aeprod11, rig 6a3383d0)

Real data only; invalid runs reported as RCAs, never faked.

✅ Proven

• F1 10k query-cap fix (live): read=42,516 http (PR adaptive_export: production AE — streaming export + write-integrity #53), read==wrote, errs=0. Re-confirmed read==wrote on this rig (broker retention caps a single window at ~7k here, so the >10k repro lives on PR adaptive_export: production AE — streaming export + write-integrity #53).
• NFR (passthrough, under load): throughput http 33 rows/s (2,120 B/s), conn 21/s, dns 5/s; footprint idle 429m CPU/27Mi → load 527m/32Mi (AE later measured 16–38Mi mem — tiny — but CPU-pinned at the old 300m limit); write fidelity write_err=0 across all cycles (read==wrote).
• DX→AE steering MECHANISM works: DX classifies, creates adaptive_attribution export windows, AE activeSet populates, streaming scanner runs whitelist-mode (it steered the live backend in run-2 before the node failed).

🔧 Hardening applied (this campaign)

• AE limits cpu=1, memory=1Gi (measured 16–38Mi; was no-mem-limit + CPU-throttled at 300m).
• PEM pemMemoryLimit=3Gi (was OOMing at the 2Gi default → PEM restarts disrupt AE capture).

🔎 Node-eviction RCA (memory was suspected)

NOT memory: nodes ran 55–60%, MemoryPressure=False. Evictions were TaintManagerEviction from node-01 going NotReady (network/kubelet-heartbeat drop), not OOM.

⛔ Datavolume reduction (DX/AE vs save-ALL) — blocked, two real causes

1. bob#140 — kubescape doesn't flag the backend's log4shell (R0011 DNS-gate), so DX never gets a backend case → it steers the wrong (low-traffic) pods (observer/postgres, indeterminate) → 0 captured → false 100%. The chain fires (marshalsec 38×) and steering works mechanically; the gap is upstream detection.
2. Rig instability — AE secret (clickhouse-dsn + pixie-api-key) reverts to placeholders on every AE roll → reduction runs (which roll AE) keep breaking. Escalated to makefile-agent.

• Modeled note: reduction = 1 − (steered-pod traffic ÷ total cluster traffic) — entirely scenario/load-dependent. A clean number needs (a) DX steering the actual attack pod (bob#140) and (b) a representative cluster load.

Harness changes committed to this PR

dead-arm guard + stale-steering clear; rows-primary reduction metric; NFR dateDiff lag fix + read==wrote; AE manifest mem-limit. Note: exp_row_reconcile.sh's ?probe= tag doesn't survive (Pixie strips query strings from req_path) — fidelity is instead evidenced by ae_reconcile read==wrote/errs=0.

[entlein]

DX-steered reduction — run 4: STEERING FIXED ✅, blocked now on cluster health (not AE)

The stage-2 fix worked — DX now steers the backend: steered-pods this run = backend ×7 (top), attacker ×3, observer/postgres ×1. marshalsec 29×, all log4j-poc pods Running (no churn). So the whole chain up to steering is proven: stage-2 → kubescape R0001(336)+R0006(24) on backend → DX ruled_in generic=MALIGNANT{execution,collection} → backend in AE activeSet.

But the DX arm still wrote 0 rows — and the AE streaming logs show why: pixieapi: stream: rpc error: code = Unavailable desc = cluster is not in a healthy state on nearly every streaming query. vizierPhase=Unhealthy (all pl pods Running, but the cluster status is degraded). The ALL/passthrough arm captured fine just minutes earlier (15,719 http rows) — the cluster degraded between arms. So the 100% is again an artifact, this time from Pixie cluster health, not AE or steering.

Chain of reduction blockers, each a distinct real cause (all now understood):

1. stale activeSet (dead pods) → fixed (clear adaptive_attribution).
2. node-01 NotReady eviction churn → rig.
3. only stage-1 fired → no R0001 → no DX case → fixed (two-stage fire); backend now steered.
4. vizierPhase=Unhealthy → streaming queries rejected → this run. Rig/cluster health.

Re-run pending a healthy cluster. AE write-path + DX steering are not the blockers.

[entlein]

Data-volume reduction — DX-steered AE vs save-ALL (basic log4shell, 2026-06-18)

Result: 99.87% row reduction / 99.33% byte reduction. DX-steered AdaptiveExport writes ~0.13% of the save-everything firehose. 5 reps each arm, reproducible, on a verified-healthy AE (aeprod14).

Mean per 2-min incident window (5 reps each)

table	ALL (firehose)	DX (steered)	row reduction
http_events	7,135	10	99.86%
dns_events	13,839	3	99.98%
conn_stats	2,762	18	99.35%
TOTAL	23,736 rows (~1.09 MB)	31 rows (~7.4 KB)	99.87% rows / 99.33% bytes

Exact conditions

• AE image: ghcr.io/k8sstormcenter/vizier-adaptive_export_image:0.14.19-aeprod14-x86_64 @ sha256:e03ce08a87aaf460de78e4f960233d479fbfa0042553aa91180adad22ab35512
• Commit: e3f7f8102 (branch ae-datavolume-dx-steering); tag release/vizier/v0.14.19-aeprod14. Carries the streaming #px:set max_output_rows_per_table=1000000 cap fix (de-biases the DX arm) on top of aeprod12 (f88650331) + aeprod13 (3e2b93a5a).
• dx-daemon: ttl.sh/dxgo-c6110ff-0609 @ sha256:14e43e0a… (dx commit c6110ff)
• Vizier (stock): ghcr.io/k8sstormcenter/vizier-*_image:0.14.17; deployed with the official pixie-io px v0.8.8 CLI.
• Stack: node-agent sbob-rc1-2026-05-16, kubescape v3.0.47, vector 0.56.0, clickhouse-server 24.8; workload log4j-chain-backend-vulnerable@sha256:72655e… + log4j-chain-attacker@sha256:c4dd5f….
• Rig: k3s v1.36.1+k3s1, cluster-id 7877e560-9f94-4b60-a2af-a0c55cfeb7f0, cloud …:443. PEM mem req/limit 2Gi/3Gi; AE limit 1Gi.
• AE env: ADAPTIVE_SKIP_APPLY=true, CONTROL_ADDR=:9100, ADAPTIVE_RECONCILE=true; ALL arm ADAPTIVE_PASSTHROUGH=true win/refresh 60/60; DX arm ADAPTIVE_WRITE_MODE=streaming win/refresh 60/60.

Protocol

Per config {ALL, DX}, 5 reps: rollout (abort if AE not Running) → restart query-broker, wait vizier Healthy → TRUNCATE all forensic_db tables → 2-min window, single full-chain fire at t=60s (JNDI/LDAP stage-1 + stage-2 in-pod command exec + DNS exfil of /etc/shadow) → measure every forensic_db table (post-truncate sizes = exactly what that config wrote) → 180s settle. No synthetic load (app-only). 4 pods steered every rep (backend/attacker/postgres/observer); chain fired 1×/rep.

Caveats

1. app-only noise model → ALL is dominated by cluster-wide infra/system traffic (kube-dns chatter + AE→CH self-traffic), not workload; the 99.9% mostly reflects "DX ignores the whole-cluster firehose and stores only the incident pods."
2. DX ≈ the attack footprint only (~31 rows); the steered pods carry no ambient load.
3. cap-clean: DX volume (≪10k) is far under the per-table cap, so it is not truncation-biased (unlike the earlier aeprod12 run where DX http hit exactly 40000 = 4×10k → 92.9% upper bound).

Full reproduction record (images, SHAs, env, harness, raw per-rep CSV, failure modes): internal biz/PoC/log4j/datavolume/june18/ (REPRODUCE.md + exp_log4j_reps.sh + aefix_per_pg.sh + log4j_reps_aeprod14.{txt,tsv}).

[entlein]

AE Non-Functional benchmark (NFR) — throughput, footprint, and end-to-end no-data-loss

Measured live on the rig under the log4shell attack-chain workload + realistic noise, two AE modes back-to-back (passthrough firehose, then DX-steered streaming). ADAPTIVE_RECONCILE=true.

End-to-end NO DATA LOSS — verified against physical ClickHouse counts

Not just AE's internal accounting — for every table we cross-checked broker read_count == AE wrote_count == actual count(*) in forensic_db:

mode	table	broker_read	AE_wrote	CH count(*)
passthrough	http_events	9,866	9,866	9,866	✅
passthrough	dns_events	42,535	42,535	42,535	✅
passthrough	conn_stats	3,856	3,856	3,856	✅
streaming	http_events	271	271	271	✅
streaming	dns_events	14	14	14	✅
streaming	conn_stats	126	126	126	✅

Zero rows lost anywhere in the broker → AE → ClickHouse path, in both modes.

Throughput

• passthrough (firehose): 56,257 rows / 156 s = 360.6 rows/s (per 2-node cluster)
• DX-streaming: 411 rows / 155 s = 2.7 rows/s — it writes only the steered-pod subset; the ~99% data-volume reduction shows up directly as throughput.

Footprint

• AE exporter: 67 MiB peak (passthrough), 38 MiB peak (streaming) — both pods summed, against a 1 GiB limit (~5% of budget). The exporter is featherweight in either mode.
• PEM: ~1.8 GiB/pod — this is Pixie's own resident cost (Stirling table store, PL_TABLE_STORE_DATA_LIMIT_MB=1228), not AE.

Takeaway: AdaptiveExport is lossless (proven against actual CH rows) and cheap (≈40–70 MiB); the only heavy resident is the stock Pixie PEM. Harness: e2e_test/adaptive_export_loadtest/harness/nfr.sh.

[ConstanzeTU]

Lets get this merged, and you code-agent please dont overwrite it anymore