``` System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: go1.6.2 linux/amd64 GitLab information Version: 11.10.0 Revision: 8a802d1c6b7 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.11 URL: https://gitlab.adcombi.com HTTP Clone URL: https://gitlab.adcombi.com/some-group/some-project.git SSH Clone URL: git@gitlab.adcombi.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 9.0.0 Repository storage paths: - default: /data/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ``` The error: ``` $ ensure_namespace + echo '$ ensure_namespace' + ensure_namespace + kubectl describe namespace master-multiadvertising-104 Error from server (Forbidden): namespaces "master-multiadvertising-104" is forbidden: User "system:serviceaccount:multiadvertising-104:multiadvertising-104-service-account" cannot get resource "namespaces" in API group "" in the namespace "master-multiadvertising-104" + kubectl create namespace master-multiadvertising-104 Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:multiadvertising-104:multiadvertising-104-service-account" cannot create resource "namespaces" in API group "" at the cluster scope ``` This used to work on `11.9.8` but after the update to `11.10`, it fails. We do a deployment to namespace per branch. The function that fails usually creates a namespace per branch for us... It's a tweaked version of the one found in the Auto-DevOps `.gitlab-ci.yml` template... ``` # example namespace: master-multiadvertising-104 function ensure_namespace() { kubectl describe namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" || kubectl create namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" } ``` It seems that `before` this function even gets triggered the deploy job already creates a namespace now using the values that got exported at the beginning of the job: ``` export KUBE_SERVICE_ACCOUNT=$'\''multiadvertising-104-service-account'\'' export KUBE_NAMESPACE=$'\''multiadvertising-104'\'' ``` And in the kubernetes dashboard, I see that these have already been created:  And by the time it hits the function those already exist with the wrong name in the wrong namespace? Something must have changed between version `11.9.8` and `11.10` Because I made numerous deploys before on `11.9.8` where this error and behaviour wasn't there. I also noticed one of the test scenarios in this MR: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352 Says: `ci_preparing_state` feature flag enabled, group cluster: `Project namespace created when deployment job runs` These are also related comments/issues: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352#note_160713703 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27352#note_161585015 https://gitlab.com/gitlab-org/gitlab-ce/issues/60971 https://gitlab.com/gitlab-org/gitlab-ce/issues/60875 ## Steps to reproduce: 1. create a project 2. create a kubernetes cluster with only the mandatory fields and tiller, ingress, cert manager, runner.. 3. notice the namespace, token, and service account made by doing: `kubectl get secrets --all-namespaces` 4. Ensure a namespace using this instead in the `.gitlab-ci.yml`: ``` function ensure_namespace() { kubectl describe namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" || kubectl create namespace "$CI_COMMIT_REF_SLUG-$KUBE_NAMESPACE" } ``` 4. Create some branches and push them. 5. See it fail in the CI It keeps using the project level credentials/ service account that was created automatically instead of the fallback cluster level credential (old behaviour) that have more permissions like it used to! ##### How can we do branch per namespace now? Is there a way to overwrite these defaults or disable default namespace creation? I tried overwriting these: `KUBE_NAMESPACE`, `KUBE_SERVICE_ACCOUNT` without any success within the CI. Please help?