Urgent Pipeline `Error from server (Forbidden): namespaces` in 11.10.1 after update from 11.9.8 Omnibus

System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.6
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.5
Go Version:	go1.6.2 linux/amd64

GitLab information
Version:	11.10.0
Revision:	8a802d1c6b7
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.11
URL:		https://gitlab.adcombi.com
HTTP Clone URL:	https://gitlab.adcombi.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.adcombi.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	9.0.0
Repository storage paths:
- default: 	/data/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

The error:

$ ensure_namespace
+ echo '$ ensure_namespace'
+ ensure_namespace
+ kubectl describe namespace master-multiadvertising-104
Error from server (Forbidden): namespaces "master-multiadvertising-104" is forbidden: User "system:serviceaccount:multiadvertising-104:multiadvertising-104-service-account" cannot get resource "namespaces" in API group "" in the namespace "master-multiadvertising-104"
+ kubectl create namespace master-multiadvertising-104
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:multiadvertising-104:multiadvertising-104-service-account" cannot create resource "namespaces" in API group "" at the cluster scope

This used to work on 11.9.8 but after the update to 11.10, it fails.

We do a deployment to namespace per branch.

The function that fails usually creates a namespace per branch for us... It's a tweaked version of the one found in the Auto-DevOps .gitlab-ci.yml template...

# example namespace: master-multiadvertising-104
function ensure_namespace() {
  kubectl describe namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" || kubectl create namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE"
}

It seems that before this function even gets triggered the deploy job already creates a namespace now using the values that got exported at the beginning of the job:

export KUBE_SERVICE_ACCOUNT=$'\''multiadvertising-104-service-account'\''
export KUBE_NAMESPACE=$'\''multiadvertising-104'\''

And in the kubernetes dashboard, I see that these have already been created:

image

And by the time it hits the function those already exist with the wrong name in the wrong namespace?

Something must have changed between version 11.9.8 and 11.10 Because I made numerous deploys before on 11.9.8 where this error and behaviour wasn't there.

I also noticed one of the test scenarios in this MR:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352

Says: ci_preparing_state feature flag enabled, group cluster: Project namespace created when deployment job runs

These are also related comments/issues:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352#note_160713703

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352#note_161585015

https://gitlab.com/gitlab-org/gitlab-ce/issues/60971

https://gitlab.com/gitlab-org/gitlab-ce/issues/60875

Steps to reproduce:

  1. create a project
  2. create a kubernetes cluster with only the mandatory fields and tiller, ingress, cert manager, runner..
  3. notice the namespace, token, and service account made by doing:

kubectl get secrets --all-namespaces

  1. Ensure a namespace using this instead in the .gitlab-ci.yml:
function ensure_namespace() {
  kubectl describe namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" || kubectl create namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE"
}
  1. Create some branches and push them.
  2. See it fail in the CI

It keeps using the project level credentials/ service account that was created automatically instead of the fallback cluster level credential (old behaviour) that have more permissions like it used to!

How can we do branch per namespace now?

Is there a way to overwrite these defaults or disable default namespace creation?

I tried overwriting these: KUBE_NAMESPACE, KUBE_SERVICE_ACCOUNT without any success within the CI.

Please help?

Edited by Khalil Gharbaoui