fix: install downstream WAF data plane in e2e CI (#263)#268
Merged
Conversation
The E2E Tests workflow is red because the chainsaw test trafficprotectionpolicy-enforce asserts Gateway waf-gw reaches Programmed=True, but the test requests gatewayClassName datum-downstream-gateway and nothing in CI programs that class. The stock prepare-infra-cluster installs only the vanilla Envoy Gateway and the NSO CRDs, never the Coraza WAF data plane the test needs. Wire the WAF data plane into the infra-cluster setup so the datum-downstream-gateway class is served end to end on kind-nso-infra. Key changes: - Add make target downstream-waf-dataplane (a prepare-infra-cluster dependency) that installs the downstream Envoy Gateway instance, the extension server, and the datum-downstream-gateway GatewayClass plus its Coraza EnvoyProxy, loading the operator image into nso-infra. - Add config/e2e-downstream overlay: reuses the config/extension-server base and resolves its mTLS placeholders with a self-signed CA chain (cert-manager Issuer issues both server and client certs; the CA Secret is mounted directly, avoiding trust-manager). Drops the ServiceMonitor and pins replicas to one for the single-node cluster. - Resolve the envoy-gateway-downstream extensionManager placeholders to the extension server FQDN and the self-signed CA Secret. - Serve the datum-downstream-gateway GatewayClass with the downstream controllerName so the test's hardcoded class matches the installed data plane; config/e2e/config.yaml is left untouched since its datum-downstream-gateway-e2e value drives the separate gateway tests. The test's stale "skipped by CI" note is corrected to reflect that prepare-infra-cluster now installs the data plane.
The trafficprotectionpolicy-enforce-serves-traffic e2e test never let waf-gw
reach Programmed=True. Two independent defects, both diverging from the
production downstream WAF wiring, kept the gateway offline.
First, the downstream Envoy Gateway watches only namespaces labelled
meta.datumapis.com/upstream-cluster-name, and its GatewayClass parametersRef
points at the EnvoyProxy in the datum-downstream-gateway namespace. That
namespace was created unlabelled, so Envoy Gateway could not see the
EnvoyProxy and left the GatewayClass unaccepted, giving every Gateway an
empty status. Production labels this namespace 0-datum-internal-anchor for
exactly this reason.
Second, the coraza-waf init container copied the WAF library from
/app/coraza-waf.so, which does not exist in coraza-waf:v1.3.0-alpha10; the
image ships it at /coraza-waf.so. The bad path made cp exit 1, the init
container CrashLoopBackOff'd, the merged Envoy pod stayed Pending, and the
Gateway reported Programmed=False with reason NoResources ("Envoy replicas
unavailable"). Production copies from /coraza-waf.so.
Key changes:
- Label the datum-downstream-gateway namespace (tools and e2e-downstream
copies) so the downstream Envoy Gateway watches it and accepts the class.
- Copy the WAF library from /coraza-waf.so, matching production.
- Pin the downstream Envoy Gateway chart to v1.7.4, the version production
runs for this control plane; CRDs still come from the base envoy-gateway
install at v1.8.1.
- Set extensionManager.maxMessageSize to 256Mi, matching production and the
extension server, so the WAF-augmented xDS response is not rejected.
Give the waf-gw Programmed assert a 3m timeout so the first cold pull of the contrib Envoy and Coraza WAF images on the kind node cannot flake it, and add failure diagnostics for the downstream WAF data plane. Key changes: - Raise the Programmed assert timeout to 3m, matching the longer timeouts the gateway-invalid-certificate suite already uses for this data plane. - Add a chainsaw catch that dumps the live Gateway, GatewayClass, and downstream Envoy Gateway and dataplane logs before cleanup tears them down. - Dump the downstream WAF data plane (GatewayClass, EnvoyProxy, controller and extension server state, certificates) from the workflow on e2e failure.
ecv
force-pushed
the
fix/263-e2e-install-downstream-waf-dataplane
branch
from
06157e0 to
1722a30
Compare
Contributor
Author
|
Heads-up while iterating on this: once the downstream WAF data plane actually installs and
So: don't chase green blindly. A green that only holds sometimes is a bug the new coverage found, which is the point of the test. |
Match the e2e overlay and production: coraza-waf:v1.3.0-alpha10 ships the WAF library at /coraza-waf.so, not /app/coraza-waf.so. The stale path made the init container's cp exit 1 and CrashLoopBackOff wherever this manifest is applied. This is the config/dev copy of the same defect fixed in config/e2e-downstream; it is currently orphaned (referenced by no Makefile/Taskfile target) but is corrected here to remove the latent trap and keep the two copies consistent.
4 tasks
scotwells
approved these changes
Jul 13, 2026
This was referenced Jul 13, 2026
ecv
added a commit
that referenced
this pull request
Jul 13, 2026
An inverted-paranoia TrafficProtectionPolicy (OWASP CRS detection below blocking) makes CRS rule 901500 fail closed and 500 every request. #252 set Accepted=False/Invalid and skipped attachment on the controller's EnvoyPatchPolicy path, but the live edge enforces coraza through the extension server, which builds per-route directives in a separate path (internal/extensionserver/cache) that #252 never touched. A grandfathered inverted policy therefore still 500'd benign traffic on the downstream gateway. Gate the extension server's directive emission on the same inverted-level check the controller uses, so an inverted policy produces no directives and the mutation layer withholds it from the data plane. A benign request is then unaffected instead of 500'ing. Key changes: - add shared TrafficProtectionPolicySpec.InvertedParanoiaLevels predicate and route both the controller resolve-error guard and the extension server emission through it - return nil directives for an inverted policy in computeCorazaDirectives so ApplyTPPRouteConfig skips the route - add API and emission-layer unit tests: inverted not emitted, valid and equal levels emitted - promote the neutralization e2e into the gating test/e2e suite and drop the now-empty test-e2e-nonblocking target Fixes #273. Refs #252, #242, #268.
4 tasks
ecv
added a commit
that referenced
this pull request
Jul 13, 2026
An inverted-paranoia TrafficProtectionPolicy (OWASP CRS detection below blocking) makes CRS rule 901500 fail closed and 500 every request. #252 set Accepted=False/Invalid and skipped attachment on the controller's EnvoyPatchPolicy path, but the live edge enforces coraza through the extension server, which builds per-route directives in a separate path (internal/extensionserver/cache) that #252 never touched. A grandfathered inverted policy therefore still 500'd benign traffic on the downstream gateway. Gate the extension server's directive emission on the same inverted-level check the controller uses, so an inverted policy produces no directives and the mutation layer withholds it from the data plane. A benign request is then unaffected instead of 500'ing. Key changes: - add shared TrafficProtectionPolicySpec.InvertedParanoiaLevels predicate and route both the controller resolve-error guard and the extension server emission through it - return nil directives for an inverted policy in computeCorazaDirectives so ApplyTPPRouteConfig skips the route - add API and emission-layer unit tests: inverted not emitted, valid and equal levels emitted - promote the neutralization e2e into the gating test/e2e suite and drop the now-empty test-e2e-nonblocking target Fixes #273. Refs #252, #242, #268.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Turns the
E2E Testsgating workflow green by installing the downstream Coraza WAF data plane on the CI infra cluster (kind-nso-infra).The chainsaw test
test/e2e/trafficprotectionpolicy-enforceasserts that Gatewaywaf-gwreachesProgrammed=True.waf-gwrequestsgatewayClassName: datum-downstream-gateway, but the stockmake prepare-infra-clusterinstalled only the vanilla Envoy Gateway (default controllerName) plus the NSO CRDs — nothing programmed thedatum-downstream-gatewayclass, so the Gateway stayedProgrammed=Unknown(status: {}) and the assert timed out on every PR.This wires the WAF data plane into the infra-cluster setup, reusing the existing
config/extension-serverandconfig/tools/envoy-gateway-downstreammanifests (previously orphaned — referenced by nothing in the Makefile/Taskfiles).Changes
make downstream-waf-dataplane(newprepare-infra-clusterdependency): installs the downstream Envoy Gateway instance (config/tools/envoy-gateway-downstream, controllerNamegateway.envoyproxy.io/datum-downstream-gateway), the extension server, and thedatum-downstream-gatewayGatewayClass + CorazaEnvoyProxy; also loads the operator image intonso-infra(newload-image-infra). Ordered aftercert-manager,envoy-gateway, anddownstream-crdsso all required CRDs exist first.config/e2e-downstream/(new overlay): references theconfig/extension-serverbase and resolves its placeholders:pki.yaml— a self-signed CA chain (selfSignedClusterIssuer→ CACertificate→ CAIssuer). The one CA issues both the extension server's TLS cert (via the cert-manager CSI driver) and Envoy Gateway's client cert; both sides trust the CA. The CASecretis mounted directly for the client-CA bundle, so trust-manager is not required (production uses a trust-managerBundle; this self-contained variant keeps the kind cluster dependency-light).waf-gateway.yaml— the CorazaEnvoyProxy(NodePort 30080/30443 for the kind hostPort) + thedatum-downstream-gatewayGatewayClass.extension-server-config.yaml— CorazarouteBaseDirectivesand branded error page enabled (the enforce test is the extension-server: branded error page injected as Envoy SubstitutionFormatString → unescaped%rejects the listener (breaks TPP Enforce) #243 regression, so the error page must be exercised).replicas: 1(single-node infra cluster) and theServiceMonitoris dropped (no Prometheus operator CRDs on the bare kind cluster).config/tools/envoy-gateway-downstream/kustomization.yaml: resolves theextensionManagerplaceholders to the extension server FQDN (envoy-gateway-extension-server.datum-downstream-gateway.svc.cluster.local) and the self-signed CASecret.GatewayClass name reconciliation
I chose the lower-blast-radius option: provision
datum-downstream-gatewayin the e2e path and leaveconfig/e2e/config.yaml(downstreamGatewayClassName: datum-downstream-gateway-e2e) untouched.downstreamGatewayClassNamedrives the NSO operator's upstream→downstream Gateway replication, which thegatewayandgateway-invalid-certificatetests depend on — they create thedatum-downstream-gateway-e2eclass inline and expect NSO to replicate onto it. The enforce test operates entirely onnso-infra(it creates the Gateway/HTTPRoute/TPP directly; the extension server reads them locally) and never flows through that config, so changing it would break the two gateway tests for no benefit. The WAF data plane installed here registersdatum-downstream-gateway, which is exactly the class the enforce test hardcodes.Validation
kustomize build config/e2e-downstream— builds clean; verified the mTLS wiring lands (tls-ca swapped to the CASecret, CSIissuer-kind: Issuer/issuer-name: envoy-gateway-extension-server, client-certissuerRefrepointed to theIssuer,SERVER_CONFIGset,ServiceMonitorremoved, no leftoverplaceholder-*refs).kustomize build config/extension-serverandconfig/e2e— still build clean.make -n prepare-infra-cluster— target graph is coherent and correctly ordered (cert-manager + CSI driver → stock EG with CRDs → downstream CRDs → image load → downstream EG instance → e2e-downstream overlay).Not validated locally (needs a CI run): I cannot run kind here, and local
kustomize --enable-helmis blocked by a helm v4 / kustomize incompatibility, so the downstream EG Helm render, the mTLS handshake between Envoy Gateway and the extension server, extension-server health, andwaf-gwactually reachingProgrammed=True+ a benign 200 on the wire are confirmed only by this PR's ownE2E Testsrun. The self-signed-CASecret-mount approach diverges from production's trust-managerBundle; if the CI run shows the CA is not resolved the way Envoy Gateway'scertificateRefexpects, the fallback is to add trust-manager + aBundleand mount the resulting ConfigMap (matching production exactly).Relationship to #248
Draft PR #248 ("Run edge e2e in CI against the real federated architecture") builds a production-fidelity Karmada-federated edge e2e — including a WAF data plane — but as a separate
test-e2e-federated.ymlworkflow that runs alongside the current one, and it is blocked on an unmerged stack (#224–#228 / #247). It does not turn the currenttest-e2e.ymlgreen. This PR is the minimal fix that unblocks the presently-red gating workflow now. Once #248 lands and the single-stacktest-e2e.ymlis retired,make downstream-waf-dataplaneandconfig/e2e-downstream/can be removed.Fixes #263. Refs #242, #248.