Some ingresses gone 404 upon upgrading to 15.6 due to "annotation contains invalid word"
## Summary
I use the bundled nginx ingress chart to serve ingresses across my entire cluster (convenience / constraints reasons--I can only have a single IP for all LoadBalancer services across the cluster)
https://gitlab.com/gitlab-org/charts/gitlab/-/commit/32923e6ad5149d263f57a2cab971eb6fa98479b8 broke my ingresses that use `location` and other common nginx directives. The affected ingresses were suddenly returning the default backend 404 page.
## Steps to reproduce
* Install 15.5.x with the bundled nginx chart set to handle all cluster ingresses (rbac.scope: false and controller.scope.enabled: false)
* Have ingresses in the cluster that use server snippet annotations, such as that from the [Nextcloud helm chart](https://github.com/nextcloud/helm)
* Install 15.6
## Configuration used
```yaml
global:
edition: ce
shell:
port: 2222
hosts:
domain: mydomain.com
ingress:
proxyBodySize: 8192m
class: nginx
# to use the same nginx ingress controller across the entire cluster and be compatible with the default Auto DevOps image,
# set this class rather than the default gitlab-nginx
# https://gitlab.com/charts/gitlab/issues/1348
# https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/assets/auto-deploy-app/values.yaml
tls:
enabled: true
appConfig:
dependencyProxy:
enabled: true
incomingEmail:
enabled: true
address: "gitlab@mydomain.com"
host: "hermod.mydomain.com"
port: 993
ssl: true
startTls: false
user: "gitlab@mydomain.com"
password:
secret: "email-password"
mailbox: inbox
idleTimeout: 60
smtp:
enabled: true
address: mail.mydomain.com
port: 25
user_name: "gitlab@mydomain.com"
## doc/installation/secrets.md#smtp-password
password:
secret: "email-password"
key: password
# domain:
authentication: "login"
starttls_auto: true
openssl_verify_mode: "peer"
email:
from: 'gitlab@mydomain.com'
display_name: My GitLab
reply_to: 'gitlab@mydomain.com'
grafana:
enabled: true
extraEnv:
PLANTUML_ENCODING: deflate
kas:
enabled: true
certmanager-issuer:
email: me@mydomain.com
certmanager:
# set default issuer and kind to a cluster issuer so it can manage certificates for ingresses in all namespaces
ingressShim:
defaultIssuerName: gitlab-cluster-issuer
defaultIssuerKind: ClusterIssuer
defaultIssuerGroup: cert-manager.io
nginx-ingress:
rbac:
scope: false
controller:
config:
use-http2: "true"
scope:
enabled: false
# we want to use this nginx-ingress deployment across the whole cluster; don't limit scope to the default namespace.
# https://gitlab.com/charts/gitlab/issues/1348
service:
externalTrafficPolicy: "Cluster"
annotations:
metallb.universe.tf/allow-shared-ip: "SharingIsCaring"
# MetalLB needs this annotation and for the externalTrafficPolicy to be set to "Cluster" in order to share a single IP to different services (so long as they use different ports)
# Other services need to have this same annotation to share the IP.
#ingressClassResource:
#name: "nginx"
# This is how ingresses figure out what controller to use
# name was hardcoded to {release name}-nginx in chart version 5.6 where it should've matched global.ingress.class
# https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3062
# The ingress resources created for the GitLab deployment and for any cert manager HTTP01 solvers are set to spec.ingressClassName = {global.ingress.class} while the IngressClass resource is set to this name, e.g. "gitlab-nginx" by default.
# Working around for now by overriding the name; this will be fixed in a future chart release and can be removed.
# Apparently the ingress.class annotation needs to be removed when the ingress has spec.ingressClassName set. You can do one or the other but not both.
registry:
ingress:
tls:
enabled: true
gitlab-runner:
install: false
# https://gitlab.com/gitlab-org/charts/gitlab/issues/1647
gitlab:
# toolbox needs storage in order to keep from being evicted
toolbox:
persistence:
enabled: true
size: 200Gi
# storageClass: temp-storage
# gitaly stores git repos (including wikis)
# gitaly's size can't be changed after deployed because it's a StatefulSet. other volumes can be expanded
gitaly:
persistence:
size: 500Gi
sidekiq:
maxReplicas: 3
# it doesn't make much sense to have more than one per node
resources:
requests:
cpu: 100m
hpa:
cpu:
targetAverageValue: 1000m
# don't scale sidekiq up until the current sidekiq pod is consistently using 1 cpu core
# minio stores registry, lfs, uploads (attached images/files in comments, etc), backups
minio:
persistence:
size: 500Gi
prometheus:
kubeStateMetrics:
enabled: true
```
## Current behavior
New default block list breaks some ingresses. Not those of GitLab itself, but others served by the ingress controller.
## Expected behavior
Not to break stuff.
Possibly should've been held back for a major chart version bump (see also https://github.com/kubernetes/ingress-nginx/issues/7939 ). But mostly just posting this as a heads-up for people like me who upgrade willy nilly without thinking about such things. I don't really care all that much either way, now that I tracked down the config setting, it was just kind of startling.
## Versions
- Chart: 6.6.0
- Platform:
- Self-hosted: kubespray v2.19.0 (calico, docker, MetalLB)
- Kubernetes: (`kubectl version`)
- Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.7", GitCommit:"42c05a547468804b2053ecf60a3bd15560362fc2", GitTreeState:"clean", BuildDate:"2022-05-24T12:30:55Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
- Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.7", GitCommit:"42c05a547468804b2053ecf60a3bd15560362fc2", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:41Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (`helm version`)
- version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
## Relevant logs
Controller logs:
```
skipping ingress namespacename/ingressname: nginx.ingress.kubernetes.io/server-snippet annotation contains invalid word location
```
issue