Error from server (Forbidden): namespaces "..." is forbidden: User "system:serviceaccount:..." cannot delete resource "namespaces" in API group ""

Summary

After the deployment of a review application (on Kubernetes with the Kubernetes executor), when the job on_stop is triggered by the merge + delete source branch action, I get an error in the output of the job when I want to delete the auto-generated namespace with this command:

kubectl delete namespace ${KUBE_NAMESPACE}

The error output is:

Error from server (Forbidden): namespaces "hello-world-18273268-preview-hello-wor-dpm6xc" is forbidden: 
User "system:serviceaccount:hello-world-18273268-preview-hello-wor-dpm6xc:hello-world-18273268-preview-hello-wor-dpm6xc-service-account" cannot delete resource "namespaces" in API group "" in the namespace "hello-world-18273268-preview-hello-wor-dpm6xc"

example of output: https://gitlab.com/tanuki-core-tutorials/civo-k3s/hello-world/-/jobs/520245337#L88

Steps to reproduce

Settings

I applied the ServiceAccount as below:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system

CI commands

Review app deployment

I deploy the review application like that

.generate-manifest: &generate-manifest
- |
  envsubst < ./kube/deploy.template.yaml > ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml
  cat -n ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml

.apply: &apply
- |
  kubectl apply -f ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml -n ${KUBE_NAMESPACE}

🎉preview-deploy:
  stage: 🚢deploy
  image: registry.gitlab.com/tanuki-workshops/tools/kube-tools:latest
  only:
    - merge_requests 
  environment:
    name: preview/${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}
    url: http://${CI_PROJECT_NAME}.${CI_COMMIT_REF_SLUG}.${DOMAIN}
    on_stop: 😢stop-preview-deploy
  script:
    - *generate-manifest
    - *apply

See: https://gitlab.com/tanuki-core-tutorials/civo-k3s/hello-world/-/blob/master/.gitlab-ci.yml#L68

🎉 And it works like a charm

Merge and delete

When I merge and delete the source branch, the job below is triggered:

.generate-manifest: &generate-manifest
- |
  envsubst < ./kube/deploy.template.yaml > ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml
  cat -n ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml

.delete: &delete
- |
  kubectl delete -f ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml -n ${KUBE_NAMESPACE}
  kubectl delete namespace ${KUBE_NAMESPACE}

😢stop-preview-deploy:
  stage: 🚢deploy
  image: registry.gitlab.com/tanuki-workshops/tools/kube-tools:latest
  only:
    - merge_requests   
  when: manual    
  environment:
    name: preview/${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}
    action: stop
  script:
    - *generate-manifest
    - *delete

See: https://gitlab.com/tanuki-core-tutorials/civo-k3s/hello-world/-/blob/master/.gitlab-ci.yml#L82

  • 😃 This command is successful: kubectl delete -f ./kube/deploy.${CI_COMMIT_SHORT_SHA}.yaml -n ${KUBE_NAMESPACE}
  • 😡 This command fails: kubectl delete namespace ${KUBE_NAMESPACE} and I get the error message below:
Error from server (Forbidden): namespaces "hello-world-18273268-preview-hello-wor-dpm6xc" is forbidden: 
User "system:serviceaccount:hello-world-18273268-preview-hello-wor-dpm6xc:hello-world-18273268-preview-hello-wor-dpm6xc-service-account" cannot delete resource "namespaces" in API group "" in the namespace "hello-world-18273268-preview-hello-wor-dpm6xc"

If I run the command kubectl get namespace, I can see my "preview" namespace hello-world-18273268-preview-hello-wor-dpm6xc

NAME                                            STATUS   AGE
default                                         Active   77m
kube-system                                     Active   77m
kube-public                                     Active   77m
kube-node-lease                                 Active   77m
gitlab-managed-apps                             Active   73m
hello-world-18273268-production-hello-usjopc    Active   43m
hello-world-18273268-preview-hello-wor-dpm6xc   Active   33m 🖐️

And I can delete it manually without any problem with this command: kubectl delete namespace hello-world-18273268-preview-hello-wor-dpm6xc

Example Project

What is the current bug behaviour?

When deleting an environment, it's not possible to delete the associated Kubernetes namespace (the namespace is generated by GitLab and I use the KUBE_NAMESPACE variable: kubectl delete namespace ${KUBE_NAMESPACE} launched from GitLab CI fails

What is the expected correct behaviour?

kubectl delete namespace ${KUBE_NAMESPACE} launched from GitLab CI is successful

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com (not tested with on-premises version)

Edited by Philippe Charrière