Conversation
…idnt work Signed-off-by: entlein <einentlein@gmail.com>
entlein
changed the title
ConstanzeTU
pushed a commit
that referenced
this pull request
Jun 20, 2026
…bit items User review 4536971862: #2 passthrough/reconcile_test.go — strengthened with two new tests that exercise the FULL chain (loop → real sink.ClickHouseHTTP → httptest CH endpoint → reconcile recorder). TestTick_ReconcileCatchesCHSilentDrop mimics CH's X-ClickHouse-Summary silent-drop shape (200 OK, written_rows=0) and asserts the loop records WroteCount=0 with a silent-drop attribution — the exact R6 (sink-layer loss) regression the instrument exists to detect. TestTick_ReconcileAttributesCHFailureCorrectly covers the 500-response branch. The pre-existing in-process-fake test stays as the wiring check. #4 pixie/pixie.go — added a long comment justifying the API-key auth choice. pixie.Client targets the Pixie CLOUD (cloudpb's PluginService), whose auth interceptor accepts pixie-api-key and rejects JWT service tokens (those are for INSIDE-cluster vizier services). The pixieapi.Adapter that talks to vizier directly already uses JWT via jwtutils.GenerateJWTForService — same pattern as cloud_connector/vizhealth/checker.go:111. So the split is intentional; flipping pixie.go to JWT would break cloud auth, not improve it. #6 pxl/queryfor — hardened escapePxL so a raw \n, \r, \t or NUL byte in Target.Pod/Target.Namespace can't terminate the PxL string literal and inject a new statement. Added TestQueryFor_RejectsInjectionInTargetFields driving QueryFor with 7 adversarial pod/namespace shapes (newline, single-quote-only, CR, backslash-escape-of-escape, NUL, tab) plus the regex_match fallback path, asserting the output line count and every statement's leading token. Extends existing TestEscapePxL_TableDriven with the new escape mappings. CodeRabbit oversights (verified each against current code): CR r3379377432 (main.go) — wrapped the control-surface listener in http.Server with Read/ReadHeader/Write/Idle timeouts so a slow client can't pin a goroutine indefinitely. CR r3379377607 (pixieapi.go) — switched direct-mode dial from pxapi.WithDisableTLSVerification (env + addr-gated, brittle) to pxapi.WithDirectTLSSkipVerify (added in PR #49 b523ce3 for the same node-IP-dial scenario). Removed the cluster.local + PX_DISABLE_TLS precondition in NewDirect; the always-skip semantics match the AE deployment shape. Refactored the obsolete env-gate test. CR r3379377645 (streaming/filter.go) — the deltaCh-close path returned without calling disarm(), leaking the timer's goroutine on shutdown. Now calls disarm() before return on chan-close. CR r3426923299 (sink/clickhouse.go Record) — capped Record at a 2s per-call context timeout. The 30s chhttp default was too long for the scanner/passthrough/controller hot paths that call Record inline; reconcile is best-effort by contract, so a stalled CH must not pin the pull loop. All 14 AE packages pass go test. arc lint: 0 Errors on PR-53 file scope.
entlein
added a commit
that referenced
this pull request
Jun 21, 2026
…bit items User review 4536971862: #2 passthrough/reconcile_test.go — strengthened with two new tests that exercise the FULL chain (loop → real sink.ClickHouseHTTP → httptest CH endpoint → reconcile recorder). TestTick_ReconcileCatchesCHSilentDrop mimics CH's X-ClickHouse-Summary silent-drop shape (200 OK, written_rows=0) and asserts the loop records WroteCount=0 with a silent-drop attribution — the exact R6 (sink-layer loss) regression the instrument exists to detect. TestTick_ReconcileAttributesCHFailureCorrectly covers the 500-response branch. The pre-existing in-process-fake test stays as the wiring check. #4 pixie/pixie.go — added a long comment justifying the API-key auth choice. pixie.Client targets the Pixie CLOUD (cloudpb's PluginService), whose auth interceptor accepts pixie-api-key and rejects JWT service tokens (those are for INSIDE-cluster vizier services). The pixieapi.Adapter that talks to vizier directly already uses JWT via jwtutils.GenerateJWTForService — same pattern as cloud_connector/vizhealth/checker.go:111. So the split is intentional; flipping pixie.go to JWT would break cloud auth, not improve it. #6 pxl/queryfor — hardened escapePxL so a raw \n, \r, \t or NUL byte in Target.Pod/Target.Namespace can't terminate the PxL string literal and inject a new statement. Added TestQueryFor_RejectsInjectionInTargetFields driving QueryFor with 7 adversarial pod/namespace shapes (newline, single-quote-only, CR, backslash-escape-of-escape, NUL, tab) plus the regex_match fallback path, asserting the output line count and every statement's leading token. Extends existing TestEscapePxL_TableDriven with the new escape mappings. CodeRabbit oversights (verified each against current code): CR r3379377432 (main.go) — wrapped the control-surface listener in http.Server with Read/ReadHeader/Write/Idle timeouts so a slow client can't pin a goroutine indefinitely. CR r3379377607 (pixieapi.go) — switched direct-mode dial from pxapi.WithDisableTLSVerification (env + addr-gated, brittle) to pxapi.WithDirectTLSSkipVerify (added in PR #49 b523ce3 for the same node-IP-dial scenario). Removed the cluster.local + PX_DISABLE_TLS precondition in NewDirect; the always-skip semantics match the AE deployment shape. Refactored the obsolete env-gate test. CR r3379377645 (streaming/filter.go) — the deltaCh-close path returned without calling disarm(), leaking the timer's goroutine on shutdown. Now calls disarm() before return on chan-close. CR r3426923299 (sink/clickhouse.go Record) — capped Record at a 2s per-call context timeout. The 30s chhttp default was too long for the scanner/passthrough/controller hot paths that call Record inline; reconcile is best-effort by contract, so a stalled CH must not pin the pull loop. All 14 AE packages pass go test. arc lint: 0 Errors on PR-53 file scope.
entlein
added a commit
that referenced
this pull request
Jun 21, 2026
…bit items User review 4536971862: #2 passthrough/reconcile_test.go — strengthened with two new tests that exercise the FULL chain (loop → real sink.ClickHouseHTTP → httptest CH endpoint → reconcile recorder). TestTick_ReconcileCatchesCHSilentDrop mimics CH's X-ClickHouse-Summary silent-drop shape (200 OK, written_rows=0) and asserts the loop records WroteCount=0 with a silent-drop attribution — the exact R6 (sink-layer loss) regression the instrument exists to detect. TestTick_ReconcileAttributesCHFailureCorrectly covers the 500-response branch. The pre-existing in-process-fake test stays as the wiring check. #4 pixie/pixie.go — added a long comment justifying the API-key auth choice. pixie.Client targets the Pixie CLOUD (cloudpb's PluginService), whose auth interceptor accepts pixie-api-key and rejects JWT service tokens (those are for INSIDE-cluster vizier services). The pixieapi.Adapter that talks to vizier directly already uses JWT via jwtutils.GenerateJWTForService — same pattern as cloud_connector/vizhealth/checker.go:111. So the split is intentional; flipping pixie.go to JWT would break cloud auth, not improve it. #6 pxl/queryfor — hardened escapePxL so a raw \n, \r, \t or NUL byte in Target.Pod/Target.Namespace can't terminate the PxL string literal and inject a new statement. Added TestQueryFor_RejectsInjectionInTargetFields driving QueryFor with 7 adversarial pod/namespace shapes (newline, single-quote-only, CR, backslash-escape-of-escape, NUL, tab) plus the regex_match fallback path, asserting the output line count and every statement's leading token. Extends existing TestEscapePxL_TableDriven with the new escape mappings. CodeRabbit oversights (verified each against current code): CR r3379377432 (main.go) — wrapped the control-surface listener in http.Server with Read/ReadHeader/Write/Idle timeouts so a slow client can't pin a goroutine indefinitely. CR r3379377607 (pixieapi.go) — switched direct-mode dial from pxapi.WithDisableTLSVerification (env + addr-gated, brittle) to pxapi.WithDirectTLSSkipVerify (added in PR #49 b523ce3 for the same node-IP-dial scenario). Removed the cluster.local + PX_DISABLE_TLS precondition in NewDirect; the always-skip semantics match the AE deployment shape. Refactored the obsolete env-gate test. CR r3379377645 (streaming/filter.go) — the deltaCh-close path returned without calling disarm(), leaking the timer's goroutine on shutdown. Now calls disarm() before return on chan-close. CR r3426923299 (sink/clickhouse.go Record) — capped Record at a 2s per-call context timeout. The 30s chhttp default was too long for the scanner/passthrough/controller hot paths that call Record inline; reconcile is best-effort by contract, so a stalled CH must not pin the pull loop. All 14 AE packages pass go test. arc lint: 0 Errors on PR-53 file scope. Signed-off-by: entlein <einentlein@gmail.com>
entlein
added a commit
that referenced
this pull request
Jun 21, 2026
) * adaptive_export: production AE — streaming export + write-integrity Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: ADAPTIVE_PASSTHROUGH firehose loop New env-gated background loop that runs the same PxL shape AE's anomaly-gated path uses, but with an empty Target (no ns/pod predicate) and over a configurable rolling window. Writes via the existing sink so the byte-shape of forensic_db rows is comparable between the PASSTHROUGH=1 phase (EVERYTHING) and the PASSTHROUGH=0 phase (AE-FILTER). One-shot A/B that yields the per-table capture fraction of the adaptive write path. - internal/passthrough/passthrough.go — Loop + Config; defaults to 30s window / 30s refresh / clickhouse.PixieTables() table list. - internal/passthrough/passthrough_test.go — 6 tests; the load-bearing one is TestLoop_EmitsEmptyTargetPxL (asserts neither df.namespace nor df.pod predicates appear in the emitted PxL). - cmd/main.go: ADAPTIVE_PASSTHROUGH + _WINDOW_SEC + _REFRESH_SEC env knobs. Adapter is constructed unconditionally when passthrough is on (joins the existing PushPixie / streaming construction path so the same pxapi grpc stream is reused). Loop is registered with the shutdown WaitGroup so SIGTERM waits for the in-flight tick. - cmd/BUILD.bazel: drop @px// load (other AE BUILD.bazel files use //bazel — sticking out as the only one with @px is a leftover from a prior gazelle run; align). Add passthrough dep. Signed-off-by: entlein <einentlein@gmail.com> * ci: dx-image workflow — build + publish dx-daemon to ghcr Stand-alone workflow that builds entlein/dx (private Active-Diagnosis Framework) into ghcr.io/k8sstormcenter/dx-daemon. Separates the dx image publish from the bazel-based vizier_release pipeline; the dx repo ships its own Dockerfile.dxd (Go cross-compile + distroless final stage) so it doesn't need to live as a submodule inside src/vizier/services/dx. Triggers: - tag push 'release/dx/v*' on this repo cuts a release build, image tag derived from the tag suffix (release/dx/v0.1.0 -> image tag 0.1.0). - workflow_dispatch lets us build any dx ref on demand with a custom tag (default: short sha of the resolved dx commit). Pulls dx via DX_ENTLEIN_PAT (already configured on the repo). Multi-arch build (linux/amd64, linux/arm64); Dockerfile.dxd cross-compiles in the native BUILDPLATFORM stage and the final stage is COPY-only, so target emulation isn't required. Signed-off-by: entlein <einentlein@gmail.com> * Revert ci: dx-image workflow — wrong repo The dx image build pipeline lives in entlein/dx itself (PR #53, branch feat/bazel-release): bazel-based with @px external pin to pixie's ae-prod tip, pushes to docker.io/entlein/dx-daemon on a release/dx/v* tag in the dx repo. The pixie-side buildx workflow this reverts duplicated that intent in the wrong repo + the wrong build system (docker buildx instead of bazel + pl_go_image macros) + the wrong registry (ghcr.io/k8sstormcenter instead of docker.io/entlein). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: unit-normalize trigger watermark cursor + load-test affordances Fixes the silent-halt bug: the trigger gated on a RAW event_time high-water-mark, so a single anomaly in a larger unit (ms/ns) drove the watermark past all real seconds rows and AE stopped processing forever (data still on Pixie). Normalize event_time to canonical nanoseconds in the poll SELECT filter+order and in the in-memory/persisted cursor, boundary-dedup, and maxSeen (normalizeEventTimeNanos + chNormEventTimeNanos). Validated at the data layer: vs a poisoned watermark the raw filter returns 0 rows, the normalized filter recovers all 60. Also adds ADAPTIVE_PUSH_REFRESH_SEC (negative = single-shot pull) for the reproducible load-test harness, an in-package trigger unit test, and an e2e hermetic load test (mock PixieQuerier, exact rows+bytes). Signed-off-by: entlein <einentlein@gmail.com> * e2e_test/adaptive_export_loadtest: AE fixture-isolation load-test harness Consolidate the adaptive_export load-test harness under src/e2e_test/ (matching vzconn_loadtest / px_cluster conventions). Control-plane experiments (E1-E4, E6, E8 sustained) are proven exactly-reproducible on a live rig; the data-plane experiments (E5, E8 data-mode) are authored and pending live validation on a vizier-registered rig (status documented in README + FINDINGS_AND_BACKLOG). - harness/: shell + python (inject, exp_control, exp_e8, stats, ...) + lib helpers - fixtures/EXPERIMENTS.md: curated kubescape_logs data-set catalog + expected outputs - k8s/: isolated sinks + per-rep generator pod (no probes) - tools/loadgen/: cleanloadgen + httpsink (docker-built test tool; .bazelignore'd pending a bazel target — lib/pq is already vendored in the module) - FINDINGS_AND_BACKLOG.md: F8 watermark-poison bug + the fix + AE backlog The AE Go unit/e2e tests live with the service (internal/{trigger,e2e}). Signed-off-by: entlein <einentlein@gmail.com> * e2e_test/adaptive_export_loadtest: document AE implied contracts (C1-C14) + diagrams Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export_loadtest: C15 write-duration contract + DX-steering diagram; gen sustained-DNS mode C15 = AE must keep re-pulling+writing an active pod until t_end or DX stop (the contract DX steers on; last week's 'wrote then stopped' is its violation). Add DX-steering sequence diagram. Generator gains SUSTAIN_SEC (distinct-DNS trickle, a Pixie-traced protocol) + configurable SETTLE_PRE_MS warm-up for fresh-pod capture; harness wires GEN_SUSTAIN_SEC/GEN_SETTLE_MS. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/trigger: update test SQL substrings for multiIf normalisation The 700821d trigger unit-normalisation wrapped event_time in a multiIf(...) inside both the WHERE filter and the ORDER BY. Three existing tests in watermark_test.go + one in clickhouse_test.go pinned the raw 'event_time >= N' substring and broke at HEAD. Update each test's expected substring to match the new normalized form (') >= <ns-scaled N>' — the closing paren of multiIf, then the value in canonical nanoseconds). Per-test ns-scaling: watermark_test.go:94 1744000000000000000 already ns -> unchanged watermark_test.go:125 InitialWatermark=42 < 1e10 sec -> * 1e9 watermark_test.go:156 InitialWatermark=7 < 1e10 sec -> * 1e9 watermark_test.go:297 event_time='5000' < 1e10 sec -> * 1e9 clickhouse_test.go:82 ref.T=1744..303e9 ns already ns -> unchanged go test ./src/vizier/services/adaptive_export/... all green. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export_loadtest: exp_control uses real now_s event_time (no future-stamp watermark poison) Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: ADAPTIVE_RECONCILE per-pull write-fidelity instrument Records one forensic_db.ae_reconcile row per data-plane pull (read_count vs wrote_count, window, ns/pod) across ALL three write paths — controller fan-out (filter), passthrough firehose, and streaming scanner — so a reconcile run localizes loss to query (R5: read<PEM) vs sink (R6: wrote<read) and quantifies re-pull dup (C8). Counts alone (write >= read) were proven insufficient. - new internal/reconcile leaf package (Row, Recorder, Nop) — no import cycle - sink.Record: CH-backed recorder (INSERT INTO forensic_db.ae_reconcile) - ae_reconcile table: schema.sql + KnownTables + OperatorOwnedTables (synced); not a pixie table (absent from PixieTables, so VerifyPixieSchema ignores it) - wired: passthrough.tick, controller.pushPixieRows (deferred, all return paths), streaming.scanner.Run; gated by ADAPTIVE_RECONCILE=true, else Nop - unit test proves read/wrote capture incl. the sink-drop read>wrote shape - fixed apply_test trailing-tables guard for the new operator table - harness: exp_row_reconcile.sh (row-level PEM<->CH), ae_vs_all.sh, exp_datavolume_extreme.sh Signed-off-by: entlein <einentlein@gmail.com> * harness: exp_pipeline_reconcile — skip empty-key rows (0 rows != LOSS 1) px -o json empty result previously printed one blank line → counted as a phantom LOSS=1. Guard: empty set → 0-byte keys file; drop all-empty-field keys. Confirmed against the controlled log4j run (backend http 14/14 exact, conn 66>=12, no loss). Signed-off-by: entlein <einentlein@gmail.com> * harness: log4shell_fire.sh — reliably fire + restart the log4j-chain log4shell Reliable BY CONSTRUCTION against bob#140 (stateful/unreliable exploit on re-fire): fresh-JVM backend (delete pod) + attacker-before-backend + the WORKING resolvable FQDN attacker.<ns>.svc.cluster.local:1389 (NOT the bare attacker-ns.svc which NXDOMAINs and gets dropped), then VERIFY the actual backend->:1389 LDAP egress in forensic_db.conn_stats and RETRY until confirmed (the validity gate). Never assumes the exploit fired. Node-side. Signed-off-by: entlein <einentlein@gmail.com> * harness: log4shell_fire.sh — detection-signal framing (Cyber Verification) Reword from offensive 'exploit' to detection-signal-generation language: this validates the kubescape->DX->AE detection chain. No logic change. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export(passthrough): precompiled + concurrent firehose, drop http2 - pxl.CompilePassthrough/Render: precompile per-table PxL once (fixed window => constant relative start_time), only the two time_ bounds are stamped per tick. Rendered output is byte-identical to QueryFor with an empty Target (TestCompilePassthrough_MatchesQueryFor), so this is a structural change, not a capture change. upid->pod/ns stays in PxL. - passthrough: tickConcurrent fans every table out at once (was a serial loop); shared pull() helper. Sink/recorder are pool/HTTP-backed and already called concurrently elsewhere. - drop http2_messages.beta from the firehose set (not materialised on every cluster => ""Table not found"" every tick); shared PixieTables/DDL lists untouched. - toggle ADAPTIVE_PASSTHROUGH_COMPILED (default on; =false reverts to the legacy serial QueryFor path). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: bazel BUILD deps for internal/reconcile + pxl compile.go Fixes "missing strict dependencies: import of .../internal/reconcile" that broke the AE image build for passthrough, sink, streaming, controller, cmd (pre-existing since the ADAPTIVE_RECONCILE commit added the package + imports without bazel deps; never CI-built). Also wires the new pxl/compile.go srcs + passthrough/pxl test srcs (compiled_test.go, reconcile_test.go, compile_test.go). - new internal/reconcile/BUILD.bazel (go_library, stdlib-only) - +//internal/reconcile dep: passthrough, sink, streaming, controller, cmd - pxl go_library +compile.go; pxl_test +compile_test.go; passthrough_test +compiled_test.go,reconcile_test.go (+reconcile dep) Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export(pxl): raise Pixie 10k result cap via #px:set query flag F1 RCA: Pixie caps every px.display at max_output_rows_per_table (default 10000, query_flags.go) — the planner add_limit_to_batch_result_sink_rule silently truncates wide firehose windows / busy pods at the READ (write path is clean: ae_reconcile shows read==wrote). Fix uses Pixie own native knob — prepend `#px:set max_output_rows_per_table=1000000` to every generated PxL (QueryFor + CompilePassthrough) so all pull paths are uncapped. Validated on rig: a 14208-row window returned 10000 (capped) vs 14298 (with flag). No pagination loop, no extra round-trips. See memory project-ae-passthrough-10k-cap. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/sink: content_type silent-drop contract suite Consolidates the recurring content_type silent-drop incident class into one default-suite test gate (6 tests, ~15ms): I1 TestContract_ContentTypeIsInt64InSchema I2 TestContract_FastEncodeContentTypeAsInt I3 TestContract_SilentDropDetected I3.b TestContract_SilentDropNotTriggeredOnSuccess I3.c TestContract_SilentDropToleratesMissingSummaryHeader I4 TestContract_HTTPEventsRoundTrip Top-of-file docstring chronicles the incident timeline so future operators can grep their way to the contract. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export_loadtest: DX-steered-vs-ALL datavolume reduction harness Measures the AdaptiveExport value prop: datavolume REDUCTION of DX-steered AE (rev-3 streaming, AE writes only DX-steered activeSet pods over the control surface) vs saving ALL data (passthrough firehose). Two arms, same fixed load, forensic_db active-part deltas (rows+bytes) per table; reduction = 1 - DX/ALL. Uses the canonical resolvable JNDI FQDN (attacker.attacker-ns.svc.cluster.local) so the chain fires + DX can classify (a malformed host → NXDOMAIN → no steer). Successor to ae_vs_all.sh, whose AE arm used the rev-2 controller gate + stale JNDI. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export_loadtest: deep AE NFR benchmark harness Measures all AE non-functional requirements under steady load on the rig: throughput (rows+bytes/sec), capture completeness (AE read vs broker count = F1 cap proof), write fidelity (read==wrote + write-error count), end-to-end freshness latency (now - max(time_) in CH), resource footprint (AE pod cpu/mem idle vs loaded), per-cycle cadence. Emits a structured report; companion to exp_dx_steering_reduction.sh. Real-data only. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export_loadtest: fix DX-reduction dead-arm (clear stale steering + live-pod guard) Run-1 reported false 100% reduction because stale adaptive_attribution windows rehydrated DEAD pods (deleted loaders) into the activeSet → AE streamed dead pods → 0 rows. Clear adaptive_attribution before the DX arm so the activeSet only gets freshly-steered LIVE pods; add a guard that prints the steered pods + marshalsec fire count + live log4j-poc pods so a dead-arm result is caught, not reported as a reduction. Signed-off-by: entlein <einentlein@gmail.com> * nfr harness: fix lag (dateDiff) + drop racy broker-pct completeness lag query used now()-DateTime64 (type error -> na); use dateDiff(second,...). Capture-completeness vs broker was window-misaligned (623% artifact) -> drop it; report tot_read vs tot_wrote (read==wrote) + errs instead. The 10k-cap/completeness proof is the dedicated F1 test (max_read>10000 vs broker for the SAME window). Signed-off-by: entlein <einentlein@gmail.com> * dx-reduction harness: report ROWS reduction (primary) + bytes (secondary) Run-2 byte-delta reduction came out negative because system.parts byte delta is compaction-noisy (merges land mid-window). Report rows reduction as primary (actual captured-row count, noise-free); keep bytes as secondary context. Signed-off-by: entlein <einentlein@gmail.com> * ae deployment: add memory limit (1Gi) + raise cpu limit to 1 core Eviction-RCA finding (PR #63 NFR campaign): AE had NO memory limit (only cpu 300m) and was CPU-pinned at 300m under concurrent passthrough. AE measured tiny (16-38Mi steady), but the raised 1M-row passthrough cap can spike, so cap at 1Gi so AE can never memory-pressure a node; raise cpu limit 300m->1 core (was throttling). NOTE node evictions were NOT AE/OOM — node-01 went NotReady (network/heartbeat); the memory consumer is PEM (1365Mi, OOMs at the 2Gi default). Signed-off-by: entlein <einentlein@gmail.com> * ae bootstrap: separate the secret from the re-applied infra bundle Root cause of the recurring "AE unauthenticated / writes 0 / crashloop" reverts: kustomization.yaml bundled adaptive_export_secrets.yaml (placeholder pixie-api-key) with the role+deployment, so EVERY infra re-apply (make log4j) clobbered the real key that ae-auth had written. Separation of concerns: remove the secret from the kustomization — infra (role+deployment) stays re-appliable; the secret holds real creds and is owned solely by `make ae-auth`, created once, never touched by infra re-applies. Secret manifest kept as a hand-applied seed-only template (documented). Signed-off-by: entlein <einentlein@gmail.com> * dx-reduction harness: fire BOTH attack stages so DX steers the backend The DX-steered arm was failing because the harness only fired stage-1 (JNDI/LDAP). That generates ldap-egress but NO kubescape R0001 → backend never flagged → DX no case → indeterminate → AE steers wrong/no pods. R0001 comes from stage-2 (post- exploitation exec). fire() now does stage-1 (JNDI) + stage-2 (whoami/shadow/token/ getent in the backend) → kubescape R0001+R0006 → DX rules backend MALIGNANT → backend enters AE activeSet → reduction is measurable. Verified live: DX evidence unexpected-spawn+sensitive-file-read → verdict ruled_in generic=MALIGNANT. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: rename whitelist→allowlist across streaming path + add DX-steering diagnostics Standing terminology rule: allowlist/blocklist, never whitelist/blacklist. Pure rename (no behavior change) of the rev-3 streaming filter: FilterModeWhitelist → FilterModeAllowlist MaxWhitelistSize → MaxAllowlistSize ADAPTIVE_STREAM_MAX_WHITELIST → ADAPTIVE_STREAM_MAX_ALLOWLIST (env) mode=whitelist log string → mode=allowlist plus all comments/identifiers/tests in streaming, activeset, cmd/main. DX-steering diagnostics (the reason DX-arm-writes-0 has been hard to RCA — we could not tell "empty ActiveSet" from "broker returned 0 rows"): - scanner: log the empty-allowlist short-circuit (was silent) so an empty ActiveSet is visible in logs, distinct from "query completed rows=0". - FilterUpdater: emitted-filter log Debug→Info so the steered pod count per ActiveSet change is visible without debug logging. NOTE: ADAPTIVE_STREAM_MAX_WHITELIST env renamed → tooling that sets the old name must switch to ADAPTIVE_STREAM_MAX_ALLOWLIST. Signed-off-by: entlein <einentlein@gmail.com> * ae(clickhouse): create forensic_db.dx_attack_graph at boot The Pixie dx_evidence_graph UI reads dx_attack_graph via px.DataFrame clickhouse_dsn, whose query template hardcodes event_time + hostname and ORDER BY event_time. A table without those columns fails 'Unknown identifier event_time'; a table created by hand (local, not via the operator) isn't globally registered. Fix: make AE own it like the other forensic tables. - schema.sql: dx_attack_graph DDL with event_time(UInt64 nanos) + hostname, edge columns, fromUnixTimestamp64Nano partition/TTL (nanos-correct). - KnownTables + OperatorOwnedTables: register it so Apply creates it at boot. - apply_test: assert last-applied DDL == last OperatorOwnedTables entry (robust to appended operator tables) instead of hardcoding trigger_watermark. go test ./.../clickhouse green. Signed-off-by: entlein <einentlein@gmail.com> * ae(clickhouse): dx_attack_graph numeric cols Int64/Float64 (px-readable) Pixie's clickhouse_dsn type mapper reads UInt8 as BOOLEAN and does not handle UInt16/UInt32/Float32 -> px fails with 'Column[N] given incorrect type' rendering the dx_evidence_graph. weight/max_severity/num_findings -> Int64, confidence -> Float64 (map cleanly to INT64/FLOAT64). Verified live: px run returns all 6 edges with every column. event_time stays UInt64 (matches kubescape_logs, which px reads). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export(streaming): add #px:set max_output_rows cap flag to scanner buildPxL The DX/streaming arm silently capped each per-table pull at Pixie's default 10000-row limit while the passthrough/ALL arm (pxl.CompilePassthrough / QueryFor) already raises it to 1,000,000 via the broker's #px:set query flag. Validated live on 6a33dac0: a single streaming http_events pull returned exactly rows=10000 (the cap). Left unfixed this UNDER-counts the DX arm and OVERSTATES the DX-vs-ALL volume reduction. Prepend the same #px:set directive to the streaming scanner's PxL so both arms are uncapped and comparable. Signed-off-by: entlein <einentlein@gmail.com> * ae(clickhouse): create dx_attack_graph_malicious view at boot Adds the rule-ins-only view (condition != '') to the canonical schema.sql, registers it in KnownTables + OperatorOwnedTables (after dx_attack_graph), and teaches DDL() to extract CREATE VIEW headers. AE now creates it on boot so the dx_evidence_graph UI's default malicious-only read is standard, not a per-rig manual step. Tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * ae(control): /dx/attack_graph ingest endpoint -> ClickHouse dx POSTs a JSON array of edges to /dx/attack_graph; AE writes them to forensic_db.dx_attack_graph via JSONEachRow (Applier.WriteAttackGraph). Wired in main.go when CONTROL_ADDR is set; 501 if the sink is unset. This is the AE half of the dx->AE->CH attack-graph write path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: convention pass — consolidate CH HTTP, drop dead code, fix stale tests PR-53 review follow-ups (see review summary in conversation): 1. License headers added to 17 src/e2e_test/adaptive_export_loadtest/ harness/*.{sh,py} scripts. Matches the convention every other src/e2e_test/*/sh in pixie already follows. 2. Dead code removed (was unreachable per golang.org/x/tools/cmd/deadcode): - internal/script/script.go: IsClickHouseScript, IsScriptForCluster, GetActions, getScriptName, getInterval, templateScript, plus the ScriptConfig and ScriptActions types. The cron-script sync flow they served was replaced by the streaming model; only the Script and ScriptDefinition types remain. - internal/pixie/pixie.go: Client.GetPresetScripts (replaced by builtinPresetScripts() inline in cmd/main.go). - internal/streaming/{supervisor,writer}.go: SupervisorStats, TableStats, Stats types + Supervisor.Stats and BatchWriter.Stats methods (no production reader). Atomic counters dropped; the existing flush log preserves the per-flush summary. 3. Stale passthrough tests fixed. TestLoop_DefaultsTablesToPixieTables and TestNew_AppliesDefaults asserted len(clickhouse.PixieTables()) == 13 but passthrough.New strips excludedTables (http2_messages.beta), yielding 12. Tests now compare against filterExcluded(clickhouse.PixieTables()) and add an extra assertion that excluded tables were not written. 4. ClickHouse HTTP client consolidation: new internal/chhttp/ package collapses three near-identical HTTP CH clients (clickhouse.Applier, sink.ClickHouseHTTP, trigger.ClickHouseWatermarkStore) into one. Centralises endpoint validation, basic-auth header, 30s default timeout, fail-loud INSERT settings (4 CH input_format knobs), and the X-ClickHouse-Summary read path. The 4 INSERT call sites in the three callers all route through chhttp.Client.Insert now; SELECT through Query or QueryStream (the latter preserves the QueryActive streaming behaviour). Net code: -200 LOC across the three callers plus a 200-LOC chhttp package with its own tests. 5. Pixie service scaffold wired into cmd/main.go: services.SetupService("adaptive-export", 50900) + services.SetupSSLClientFlags() + services.PostFlagSetupAndParse() + services.SetupServiceLogging(). Matches the pattern every other pixie Go service uses. CheckServiceFlags() is deliberately skipped (AE does not run a TLS gRPC server). Existing AE env-var reads (ADAPTIVE_*) are untouched and still authoritative for tuning knobs. All 14 adaptive_export internal packages pass go test (1 new chhttp, 13 unchanged), including the 3 differential oracle tests in pxl/compile_test.go. arc lint OKAY for everything except 3 pre-existing findings unrelated to this commit (loadgen has its own go.mod; one QF1001 De Morgan's-law nit in reconcile_test.go from c9f19b6). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: restore executable bits on harness scripts (post-header) Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: lint pass + restore @px load prefix in cmd/BUILD.bazel User flagged on review 4536971862: - cmd/BUILD.bazel:18 dropped the '@px' external-workspace prefix on pl_build_system.bzl load. Restored — the standalone AE build pulls pixie as @px and needs the qualifier. Lint cleanup (PR-53 scope, no production code touched outside renames): - chhttp/chhttp_test.go: errcheck on two w.Write calls + gofumpt on the gotSettings declaration. - passthrough/reconcile_test.go: staticcheck QF1001 (De Morgan's law) on the conn_stats sink-drop assertion. - script/script.go: rename ScriptId -> ScriptID (ST1003). Propagated to pixie/pixie.go and cmd/main.go callsites. - internal/e2e/BUILD.bazel and internal/trigger/BUILD.bazel: gazelle drift — adding loadtest_test.go and clickhouse_internal_test.go to srcs. - k8s/00-sinks.yaml + gen-pod.tmpl.yaml: yamllint compliance — document-start marker, dedent sequence items per .yamllint indent-sequences=false, tighten flow-mapping spaces, collapse multi-space after commas. YAML semantics unchanged. - harness/stats.py: flake8 E501 — split a long line into two. Final arc lint state on PR-53 file scope: 0 Errors, 14 Warnings (SHELLCHECK SC2155/SC2086 in pre-existing harness scripts from ae7b86f, not introduced or modified by this commit). Pre-existing loadgen typecheck failures (separate go.mod) are unaffected. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: address user review #2 #4 #6 + 4 outstanding CodeRabbit items User review 4536971862: #2 passthrough/reconcile_test.go — strengthened with two new tests that exercise the FULL chain (loop → real sink.ClickHouseHTTP → httptest CH endpoint → reconcile recorder). TestTick_ReconcileCatchesCHSilentDrop mimics CH's X-ClickHouse-Summary silent-drop shape (200 OK, written_rows=0) and asserts the loop records WroteCount=0 with a silent-drop attribution — the exact R6 (sink-layer loss) regression the instrument exists to detect. TestTick_ReconcileAttributesCHFailureCorrectly covers the 500-response branch. The pre-existing in-process-fake test stays as the wiring check. #4 pixie/pixie.go — added a long comment justifying the API-key auth choice. pixie.Client targets the Pixie CLOUD (cloudpb's PluginService), whose auth interceptor accepts pixie-api-key and rejects JWT service tokens (those are for INSIDE-cluster vizier services). The pixieapi.Adapter that talks to vizier directly already uses JWT via jwtutils.GenerateJWTForService — same pattern as cloud_connector/vizhealth/checker.go:111. So the split is intentional; flipping pixie.go to JWT would break cloud auth, not improve it. #6 pxl/queryfor — hardened escapePxL so a raw \n, \r, \t or NUL byte in Target.Pod/Target.Namespace can't terminate the PxL string literal and inject a new statement. Added TestQueryFor_RejectsInjectionInTargetFields driving QueryFor with 7 adversarial pod/namespace shapes (newline, single-quote-only, CR, backslash-escape-of-escape, NUL, tab) plus the regex_match fallback path, asserting the output line count and every statement's leading token. Extends existing TestEscapePxL_TableDriven with the new escape mappings. CodeRabbit oversights (verified each against current code): CR r3379377432 (main.go) — wrapped the control-surface listener in http.Server with Read/ReadHeader/Write/Idle timeouts so a slow client can't pin a goroutine indefinitely. CR r3379377607 (pixieapi.go) — switched direct-mode dial from pxapi.WithDisableTLSVerification (env + addr-gated, brittle) to pxapi.WithDirectTLSSkipVerify (added in PR #49 b523ce3 for the same node-IP-dial scenario). Removed the cluster.local + PX_DISABLE_TLS precondition in NewDirect; the always-skip semantics match the AE deployment shape. Refactored the obsolete env-gate test. CR r3379377645 (streaming/filter.go) — the deltaCh-close path returned without calling disarm(), leaking the timer's goroutine on shutdown. Now calls disarm() before return on chan-close. CR r3426923299 (sink/clickhouse.go Record) — capped Record at a 2s per-call context timeout. The 30s chhttp default was too long for the scanner/passthrough/controller hot paths that call Record inline; reconcile is best-effort by contract, so a stalled CH must not pin the pull loop. All 14 AE packages pass go test. arc lint: 0 Errors on PR-53 file scope. Signed-off-by: entlein <einentlein@gmail.com> * test(harness): consolidate to one run-picture + e2e CI workflow Finish the harness consolidation #53 started: adopt exp_matrix.sh (canonical ALL-vs-DX reduction matrix) + nfr.sh (throughput/mem/verdict-latency) as the single runners; cut the overlapping variants (ae_vs_all, exp_datavolume_extreme, exp_dx_steering_reduction, exp_ae_nfr_benchmark, exp_pipeline_reconcile) and the superseded standalone setup (deploy_ae, build_gen_image). README rewritten as the single 'how to run' source: two families — fixture-isolation (run.sh + E-series) and live-attack e2e (log4shell_fire → exp_matrix → nfr → exp_row_reconcile). Add e2e_log4shell_soc.yaml: k3s + full SOC stack (Pixie/kubescape/ClickHouse/AE/dx via k8sstormcenter/soc) on the oracle runner; asserts every canonical harness script runs + dx rules in; profiles dx in real life (pprof CPU/heap + verdict latency, uploaded). Uses existing repo secrets. Signed-off-by: entlein <einentlein@gmail.com> * revert(bazel): drop stray buildifier attribute-reorder in stirling container_images BUILD This file is unrelated to adaptive_export — the only change was buildifier alphabetizing container_type/bazel_sdk_versions (no functional change). Restore main's version to keep #53's diff to real AE changes. Signed-off-by: entlein <einentlein@gmail.com> * ci: fix run-genfiles + run-container-lint on PR 53 run-genfiles: the PR had reordered the kwargs in stirling/.../container_images/ BUILD.bazel alphabetically (bazel_sdk_versions before container_type) — a local buildifier or gazelle drift from when the file was first committed. CI gazelle wants the original order (container_type first), so the diff loop fails. Restored to origin/main's ordering. Local gazelle disagrees with CI's expected output (tooling-version drift); CI is authoritative. run-container-lint: two findings. 1. staticcheck QF1001 (De Morgan's law) in pxl/queryfor_test.go:318. Rewrote the !(a || b || c || d) negated disjunction as the equivalent !a && !b && !c && !d. Test behaviour unchanged; verified locally with TestQueryFor_RejectsInjection. 2. golangci-lint typechecking failed on src/e2e_test/adaptive_export_loadtest/tools/loadgen/cmd/{cleanloadgen, httpsink}/main.go because that subtree carries its own go.mod and does not resolve as a package under the root px.dev/pixie module. Added the loadgen subtree to .arclint's exclude list. Same fix the existing entries for other-module subtrees apply. Local 'arc lint' on the PR-53 file scope: 0 Errors, 14 SHELLCHECK Warnings + 26 SHELLCHECK Advice (all pre-existing in ae7b86f harness scripts; not introduced by this PR). Signed-off-by: entlein <einentlein@gmail.com> * ci: apply gazelle's actual kwarg order to stirling container_images run-genfiles CI re-failed after f244ffc reverted this file to main's ordering — turns out gazelle on this repo IS alphabetizing the kwargs (bazel_sdk_versions before container_type), and CI runs 'gazelle fix' then 'git diff' to catch any drift. So main's ordering is no longer gazelle-stable; the file has to match gazelle's preference, not main's. Verified locally with 'bazel run //:gazelle -- fix'; the file is now idempotent (a second gazelle run produces no diff). Signed-off-by: entlein <einentlein@gmail.com> * ci: fix container-lint Errors on e2e workflow + new harness scripts run-container-lint re-failed after the run-genfiles fix because two files added on this branch (a03aa15) had unfixed lint errors: .github/workflows/e2e_log4shell_soc.yaml — 5 yamllint Errors: - 1 indentation: list items under steps: must be parent-aligned per the repo's .yamllint config (indent-sequences: false), not 2-indented. Dedented every step item + its run: block by 2 spaces. - 4 line-length (>120 chars): split the long kubectl-set-image, the long grep-detection gate, the curl pprof URL, and the verdict-latency grep across continuation lines. Semantics unchanged. src/e2e_test/adaptive_export_loadtest/harness/{exp_matrix.sh, nfr.sh} — missing Apache headers. Applied via arc lint --apply-patches; exec bits restored. Local arc lint on PR-53 file scope: 0 Errors, 14 SHELLCHECK Warnings + 26 Advice (all pre-existing in harness scripts, unchanged). Signed-off-by: entlein <einentlein@gmail.com> * ci: silence 6 SHELLCHECK Warnings to clear container-lint exit code arc lint --apply-patches exits non-zero on Warning level too. Resolved each: replaced unused 'for i/t in ...' loop vars with '_', split a SC2155 declare-and-assign, dropped two never-referenced hip/pip assignments. Signed-off-by: entlein <einentlein@gmail.com> --------- Signed-off-by: entlein <einentlein@gmail.com> Co-authored-by: Entlein <eineintlein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: pixie-agent <noreply@local>
ConstanzeTU
pushed a commit
that referenced
this pull request
Jul 4, 2026
…-assets regression guard dx-agent (entlein/dx#119) on Ubuntu 24.04: - #2: dx-standalone.env had 'DX_CATALOG_DIR=/etc/dx/catalog # ...' — systemd EnvironmentFile keeps the inline comment IN the value (daemon logged dir='/etc/dx/catalog # ...'). Moved every comment to its own line + a warning header. - test/verify-assets.sh: regression guard — CI mode asserts no inline comments in the .env + the launcher ExecStart; asset mode validates a downloaded bundle (#1 top symlink relative/resolves, #4 packaged headers present). All PASS. standalone-pem.service already execs the /usr/local/bin/standalone-pem launcher (native), so #1's dangling top symlink didn't affect the unit — but the tarball is repackaged with a relative symlink anyway (entlein/dx release asset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
need to find a place for these guys