Some ingresses gone 404 upon upgrading to 15.6 due to "annotation contains invalid word"
Summary
I use the bundled nginx ingress chart to serve ingresses across my entire cluster (convenience / constraints reasons--I can only have a single IP for all LoadBalancer services across the cluster)
32923e6a broke my ingresses that use location
and other common nginx directives. The affected ingresses were suddenly returning the default backend 404 page.
Steps to reproduce
- Install 15.5.x with the bundled nginx chart set to handle all cluster ingresses (rbac.scope: false and controller.scope.enabled: false)
- Have ingresses in the cluster that use server snippet annotations, such as that from the Nextcloud helm chart
- Install 15.6
Configuration used
global:
edition: ce
shell:
port: 2222
hosts:
domain: mydomain.com
ingress:
proxyBodySize: 8192m
class: nginx
# to use the same nginx ingress controller across the entire cluster and be compatible with the default Auto DevOps image,
# set this class rather than the default gitlab-nginx
# https://gitlab.com/charts/gitlab/issues/1348
# https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/assets/auto-deploy-app/values.yaml
tls:
enabled: true
appConfig:
dependencyProxy:
enabled: true
incomingEmail:
enabled: true
address: "gitlab@mydomain.com"
host: "hermod.mydomain.com"
port: 993
ssl: true
startTls: false
user: "gitlab@mydomain.com"
password:
secret: "email-password"
mailbox: inbox
idleTimeout: 60
smtp:
enabled: true
address: mail.mydomain.com
port: 25
user_name: "gitlab@mydomain.com"
## doc/installation/secrets.md#smtp-password
password:
secret: "email-password"
key: password
# domain:
authentication: "login"
starttls_auto: true
openssl_verify_mode: "peer"
email:
from: 'gitlab@mydomain.com'
display_name: My GitLab
reply_to: 'gitlab@mydomain.com'
grafana:
enabled: true
extraEnv:
PLANTUML_ENCODING: deflate
kas:
enabled: true
certmanager-issuer:
email: me@mydomain.com
certmanager:
# set default issuer and kind to a cluster issuer so it can manage certificates for ingresses in all namespaces
ingressShim:
defaultIssuerName: gitlab-cluster-issuer
defaultIssuerKind: ClusterIssuer
defaultIssuerGroup: cert-manager.io
nginx-ingress:
rbac:
scope: false
controller:
config:
use-http2: "true"
scope:
enabled: false
# we want to use this nginx-ingress deployment across the whole cluster; don't limit scope to the default namespace.
# https://gitlab.com/charts/gitlab/issues/1348
service:
externalTrafficPolicy: "Cluster"
annotations:
metallb.universe.tf/allow-shared-ip: "SharingIsCaring"
# MetalLB needs this annotation and for the externalTrafficPolicy to be set to "Cluster" in order to share a single IP to different services (so long as they use different ports)
# Other services need to have this same annotation to share the IP.
#ingressClassResource:
#name: "nginx"
# This is how ingresses figure out what controller to use
# name was hardcoded to {release name}-nginx in chart version 5.6 where it should've matched global.ingress.class
# https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3062
# The ingress resources created for the GitLab deployment and for any cert manager HTTP01 solvers are set to spec.ingressClassName = {global.ingress.class} while the IngressClass resource is set to this name, e.g. "gitlab-nginx" by default.
# Working around for now by overriding the name; this will be fixed in a future chart release and can be removed.
# Apparently the ingress.class annotation needs to be removed when the ingress has spec.ingressClassName set. You can do one or the other but not both.
registry:
ingress:
tls:
enabled: true
gitlab-runner:
install: false
# https://gitlab.com/gitlab-org/charts/gitlab/issues/1647
gitlab:
# toolbox needs storage in order to keep from being evicted
toolbox:
persistence:
enabled: true
size: 200Gi
# storageClass: temp-storage
# gitaly stores git repos (including wikis)
# gitaly's size can't be changed after deployed because it's a StatefulSet. other volumes can be expanded
gitaly:
persistence:
size: 500Gi
sidekiq:
maxReplicas: 3
# it doesn't make much sense to have more than one per node
resources:
requests:
cpu: 100m
hpa:
cpu:
targetAverageValue: 1000m
# don't scale sidekiq up until the current sidekiq pod is consistently using 1 cpu core
# minio stores registry, lfs, uploads (attached images/files in comments, etc), backups
minio:
persistence:
size: 500Gi
prometheus:
kubeStateMetrics:
enabled: true
Current behavior
New default block list breaks some ingresses. Not those of GitLab itself, but others served by the ingress controller.
Expected behavior
Not to break stuff.
Possibly should've been held back for a major chart version bump (see also https://github.com/kubernetes/ingress-nginx/issues/7939 ). But mostly just posting this as a heads-up for people like me who upgrade willy nilly without thinking about such things. I don't really care all that much either way, now that I tracked down the config setting, it was just kind of startling.
Versions
- Chart: 6.6.0
- Platform:
- Self-hosted: kubespray v2.19.0 (calico, docker, MetalLB)
- Kubernetes: (
kubectl version
)- Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.7", GitCommit:"42c05a547468804b2053ecf60a3bd15560362fc2", GitTreeState:"clean", BuildDate:"2022-05-24T12:30:55Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
- Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.7", GitCommit:"42c05a547468804b2053ecf60a3bd15560362fc2", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:41Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)- version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
Relevant logs
Controller logs:
skipping ingress namespacename/ingressname: nginx.ingress.kubernetes.io/server-snippet annotation contains invalid word location