Description:
Customizing EP deployment name in GatewayNamespace mode throws error and the EP pods won't start. Hint: the error message references the default deployment name, not the customized one: failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93
Repro steps:
Chart values
config:
envoyGateway:
provider:
kubernetes:
deploy:
type: GatewayNamespaceManifest
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gateway
namespace: bogusita-lab
spec:
gatewayClassName: envoy-gateway-default
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: envoy-gateway-default
listeners:
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
group: ""
name: gateway-certificate
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: envoy-gateway-default
namespace: bogusita-lab
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
# Configure the managed Envoy Proxy Deployment to be named 'gateway'
name: gateway
# Override resource requests and limits for the managed Envoy Proxy container 'envoy' to comply with ResourceQuota requirements
container:
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 512Mi
# Patch a memory limit for the managed Envoy Proxy container 'shutdown-manager' to comply with ResourceQuota requirements
patch:
type: StrategicMerge
value:
spec:
template:
spec:
containers:
- name: shutdown-manager
resources:
limits:
memory: 32Mi
# Enable HPA for the managed Envoy Proxy
envoyHpa:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 60
type: Utilization
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 180
envoyService:
# Configure the managed Envoy Proxy Service to be cluster-internal
type: ClusterIP
# Configure the managed Envoy Proxy Service to be named 'gateway'
name: gateway
# Enable PodDisruptionBudget for the managed Envoy Proxy
envoyPDB:
maxUnavailable: 50%Environment:
Running on a completely fresh installation of K8s + EG
EG version: v0.0.0-latest
K8s version: 1.31.7
K8s provider: AKS
Logs:
EG
2025-05-12T09:32:39.107Z ERROR infrastructure runner/runner.go:116 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.108Z INFO infrastructure runner/runner.go:100 received an update {"runner": "infrastructure"}
2025-05-12T09:32:39.108Z ERROR watchable message/watchutil.go:82 observed an error {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.209Z ERROR infrastructure runner/runner.go:116 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-lab-gateway-4c2a784c: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.210Z INFO infrastructure runner/runner.go:100 received an update {"runner": "infrastructure"}
2025-05-12T09:32:39.210Z ERROR watchable message/watchutil.go:82 observed an error {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-lab-gateway-4c2a784c: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.309Z ERROR infrastructure runner/runner.go:116 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-prd-gateway-1daa1257: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.309Z INFO infrastructure runner/runner.go:100 received an update {"runner": "infrastructure"}
2025-05-12T09:32:39.309Z ERROR watchable message/watchutil.go:82 observed an error {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-prd-gateway-1daa1257: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.578Z INFO infrastructure runner/runner.go:100 received an update {"runner": "infrastructure"}
2025-05-12T09:32:39.624Z ERROR infrastructure runner/runner.go:116 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
2025-05-12T09:32:39.624Z ERROR watchable message/watchutil.go:82 observed an error {"runner": "infrastructure", "error": "failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93: an illegal change in a custom label of EnvoyProxy is detected when updating bogusita-lab/gateway. The custom label config of deployment in EnvoyProxy, which is initiated with the envoy gateway of v1.1 or earlier, is immutable. Please recreate an envoy proxy with a new custom label if you need to change the custom label. This issue won't happen with the envoy proxy resource initialized by the envoygateway v1.2 or later"}
EP
$ kubectl -n bogusita-lab get pods
NAME READY STATUS RESTARTS AGE
gateway-6ddcd956c8-brpfn 0/2 CrashLoopBackOff 7 (77s ago) 7m10s
gateway-6ddcd956c8-z82k7 0/2 CrashLoopBackOff 7 (84s ago) 7m13s
$
$
$ kubectl -n bogusita-lab logs gateway-6ddcd956c8-brpfn
Defaulted container "envoy" out of: envoy, shutdown-manager
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.filters.http.credential_injector.v3.CredentialInjector' is contained in proto file 'envoy/extensions/filters/http/credential_injector/v3/credential_injector.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.http.injected_credentials.generic.v3.Generic' is contained in proto file 'envoy/extensions/http/injected_credentials/generic/v3/generic.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.filters.http.credential_injector.v3.CredentialInjector' is contained in proto file 'envoy/extensions/filters/http/credential_injector/v3/credential_injector.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.filters.http.credential_injector.v3.CredentialInjector' is contained in proto file 'envoy/extensions/filters/http/credential_injector/v3/credential_injector.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.http.injected_credentials.generic.v3.Generic' is contained in proto file 'envoy/extensions/http/injected_credentials/generic/v3/generic.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.673][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:39] message 'envoy.extensions.filters.http.credential_injector.v3.CredentialInjector' is contained in proto file 'envoy/extensions/filters/http/credential_injector/v3/credential_injector.proto' marked as work-in-progress. API features marked as work-in-progress are not considered stable, are not covered by the threat model, are not supported by the security team, and are subject to breaking changes. Do not use this feature without understanding each of the previous points.
[2025-05-12 11:37:42.718][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:42.860][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:43.666][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:43.837][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:47.193][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:54.317][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:37:57.765][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster
[2025-05-12 11:38:06.314][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:38:12.769][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
[2025-05-12 11:38:17.913][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:38:45.817][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:38:53.646][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:38:55.870][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:38:58.769][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:39:18.683][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:39:41.305][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:39:44.015][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:39:50.273][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:40:07.230][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:40:13.234][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
[2025-05-12 11:40:34.666][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 2, failed to validate token: pod [gateway-6ddcd956c8-brpfn] not found in cache{via_upstream}
Description:
Customizing EP deployment name in GatewayNamespace mode throws error and the EP pods won't start. Hint: the error message references the default deployment name, not the customized one:
failed to create or update deployment bogusita-lab/envoy-bogus-dev-gateway-1fb8ea93Repro steps:
Chart values
Manifest
Environment:
Running on a completely fresh installation of K8s + EG
EG version:
v0.0.0-latestK8s version: 1.31.7
K8s provider: AKS
Logs:
EG
EP