Unable to upgrade ingress from integration page
Summary
I”m trying to upgrade the ingress from the integration page. The previous ingress chart version is nginx-ingress-1.22.1
- the new version is nginx-ingress-1.29.7
. This upgrade is unsuccessful. Additionally, there is no way to fix this from the UI (the ingress cannot be removed and re-added).
Steps to reproduce
I’m upgrading by switching the ModSecurity Web Application Firewall to Enabled and clicking Save. It seems to be trying - the K8s deployment shows the new version of the chart. The install-ingress pod keeps erroring out though.
Example Project
https://gitlab.com/groups/gitlab-org/gitlab-services/-/clusters/67868?tab=apps
What is the current bug behavior?
- I keep getting
Something went wrong while trying to save your settings. Please try again.
- Looking at the install-ingress pod logs, I see
UPGRADE FAILED: kind ServiceAccount with the name "ingress-nginx-ingress-backend" already exists in the cluster
- Manually deleting that service account and running again, I see
Error: Service "ingress-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable
What is the expected correct behavior?
The ingress is upgraded without errors.
Relevant logs and/or screenshots
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.3 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
+ helm upgrade ingress stable/nginx-ingress --install --reset-values --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem --version 1.29.7 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/ingress/config/values.yaml
Error: UPGRADE FAILED: kind ServiceAccount with the name "ingress-nginx-ingress-backend" already exists in the cluster and wasn't defined in the previous release. Before upgrading, please either delete the resource from the cluster or remove it from the chart
UPGRADE FAILED
Error: kind ServiceAccount with the name "ingress-nginx-ingress-backend" already exists in the cluster and wasn't defined in the previous release. Before upgrading, please either delete the resource from the cluster or remove it from the chart
$ kubectl delete serviceaccount ingress-nginx-ingress-backend --namespace gitlab-managed-apps
serviceaccount "ingress-nginx-ingress-backend" deleted
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.3 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
+ helm upgrade ingress stable/nginx-ingress --install --reset-values --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key
/data/helm/ingress/config/key.pem --version 1.29.7 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/ingress/config/values.yaml
UPGRADE FAILED
Error: Service "ingress-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable
Error: UPGRADE FAILED: Service "ingress-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)