Summary
The extension-server injects the branded error page into Envoy's local_reply_config as a SubstitutionFormatString / TextFormatSource. Envoy parses that string as a format string, so every literal % in the HTML is treated as a command operator. The embedded default page contains unescaped % characters, so Envoy rejects the entire listener — and with the downstream Envoy Gateway running failOpen: false, the rejected xDS update is dropped fleet-wide, leaving no listener. Result: 100% traffic failure on any TrafficProtectionPolicy in Enforce mode that falls back to the embedded page.
Discovered while reproducing #242 on a full local kind WAF stack.
Evidence
Envoy on the merged downstream gateway:
[warning][config] delta config for type.googleapis.com/envoy.config.listener.v3.Listener
rejected: Error adding/updating listener(s) tcp-80: Incorrect configuration: <!DOCTYPE html>
...
. Couldn't find valid command at position 330
/listeners then shows only the admin/stats listeners (no tcp-80), and clients get curl: (52) Empty reply from server. The backend is healthy (direct hit returns 200). Removing the TPP restores the listener.
Byte 330 of internal/extensionserver/assets/error-5xx-default.html is the % in:
html, body { height: 100%; margin: 0; }The embedded page has 12 single % and zero %% escapes.
Root cause
internal/extensionserver/mutate/localreply.go (buildLocalReplyConfig):
BodyFormatOverride: &corev3.SubstitutionFormatString{
ContentType: cfg.ContentType,
Format: &corev3.SubstitutionFormatString_TextFormatSource{
TextFormatSource: &corev3.DataSource{
Specifier: &corev3.DataSource_InlineString{InlineString: cfg.BodyHTML},
},
},
},%RESPONSE_CODE% is the intended (valid) substitution, but any bare % in the body is not. error-5xx-default.html does not escape its literal %. The infra-mounted override page (apps/envoy-error-pages/.../error-5xx.html) does escape them (width: 100%%), which is why prod hasn't been taken down by this specific path — but any gateway without the envoy-error-pages ConfigMap falls back to the malformed embedded page.
Fix
- Escape literal
% → %% in error-5xx-default.html (and any other content fed into a SubstitutionFormatString).
- Validate/round-trip the assembled
local_reply_config at extension-server startup so a malformed body is caught before it ever reaches Envoy, rather than silently NACKing xDS in the data plane.
Also worth confirming
While validating the fix locally, after mounting the escaped override page and restarting the extension-server (which logged using error-page body override ... bytes=13047), the pushed listener config still carried the embedded page and stayed rejected. Most likely Envoy Gateway caching the pre-override translation (re-applying an unchanged TPP doesn't retrigger the xDS hook), but confirm the loaded override actually reaches InjectLocalReplyConfig and isn't shadowed by the embedded default.
Scope / impact
Any TrafficProtectionPolicy in Enforce mode on a gateway using the embedded default error page → the WAF listener never programs → total outage on that gateway (not a WAF rule match). Sub-issue of #242.
Repro notes
Reproduced on Envoy contrib-v1.35.0 (the tag in config/dev/downstream_resources/downstream-gateway.yaml); edge runs contrib-v1.37.1, where the wire symptom differs (reported 500 + content-length + 0 body + transfer closed vs. the listener-reject "empty reply" here) but is consistent with the same unescaped-% root. See #242 for the full local-stack writeup.
Summary
The extension-server injects the branded error page into Envoy's
local_reply_configas aSubstitutionFormatString/TextFormatSource. Envoy parses that string as a format string, so every literal%in the HTML is treated as a command operator. The embedded default page contains unescaped%characters, so Envoy rejects the entire listener — and with the downstream Envoy Gateway runningfailOpen: false, the rejected xDS update is dropped fleet-wide, leaving no listener. Result: 100% traffic failure on anyTrafficProtectionPolicyinEnforcemode that falls back to the embedded page.Discovered while reproducing #242 on a full local kind WAF stack.
Evidence
Envoy on the merged downstream gateway:
/listenersthen shows only the admin/stats listeners (notcp-80), and clients getcurl: (52) Empty reply from server. The backend is healthy (direct hit returns 200). Removing the TPP restores the listener.Byte 330 of
internal/extensionserver/assets/error-5xx-default.htmlis the%in:The embedded page has 12 single
%and zero%%escapes.Root cause
internal/extensionserver/mutate/localreply.go(buildLocalReplyConfig):%RESPONSE_CODE%is the intended (valid) substitution, but any bare%in the body is not.error-5xx-default.htmldoes not escape its literal%. The infra-mounted override page (apps/envoy-error-pages/.../error-5xx.html) does escape them (width: 100%%), which is why prod hasn't been taken down by this specific path — but any gateway without theenvoy-error-pagesConfigMap falls back to the malformed embedded page.Fix
%→%%inerror-5xx-default.html(and any other content fed into aSubstitutionFormatString).local_reply_configat extension-server startup so a malformed body is caught before it ever reaches Envoy, rather than silently NACKing xDS in the data plane.Also worth confirming
While validating the fix locally, after mounting the escaped override page and restarting the extension-server (which logged
using error-page body override ... bytes=13047), the pushed listener config still carried the embedded page and stayed rejected. Most likely Envoy Gateway caching the pre-override translation (re-applying an unchanged TPP doesn't retrigger the xDS hook), but confirm the loaded override actually reachesInjectLocalReplyConfigand isn't shadowed by the embedded default.Scope / impact
Any
TrafficProtectionPolicyinEnforcemode on a gateway using the embedded default error page → the WAF listener never programs → total outage on that gateway (not a WAF rule match). Sub-issue of #242.Repro notes
Reproduced on Envoy
contrib-v1.35.0(the tag inconfig/dev/downstream_resources/downstream-gateway.yaml); edge runscontrib-v1.37.1, where the wire symptom differs (reported500+content-length+ 0 body +transfer closedvs. the listener-reject "empty reply" here) but is consistent with the same unescaped-%root. See #242 for the full local-stack writeup.