gitlab provided $KUBECONFIG in ci is not following $KUBE_NAMESPACE

Summary

kubernetes account provided in KUBECONFIG is not in the right namespace when setting namespace with KUBE_NAMESPACE in CI settings

Steps to reproduce

Add a RBAC enabled kubernetes cluster to your project

Set up $KUBE_NAMESPACE to whatever existing namespace in your kubernetes cluster.

set up a CI stage with an environment :

and try to apply a file in the cluster : kubectl apply -f ./file.yml --namespace $KUBE_NAMESPACE

What is the current bug behavior?

user provided in $KUBECONFIG has no edit right in the $KUBE_NAMESPACE

and $KUBECONFIG content :

What is the expected correct behavior?

namespace should be $KUBE_NAMESPACE and this token should have edit rights in $KUBE_NAMESPACE

Relevant logs and/or screenshots

see previous screenshots

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

This bug happens on GitLab.com

Results of GitLab application Check

This bug happens on GitLab.com

Possible fixes

changed the description

added Category:Kubernetes Management devopsconfigure [DEPRECATED] groupconfigure [DEPRECATED] severity3 typebug labels

@teo.goddet thanks for this issue and giving all of these details. It makes it much easier for us to understand what's going on.

Basically I believe the problem here is that GitLab is just not designed to have people configuring the KUBE_NAMESPACE CI variable. The parts of GitLab that create namespaces and service accounts for each CI job were not built to look for this user configured variable but rather they only look at the namespace set when creating the cluster which is not possible to set in all cases. You'll have some cases where this works to an extent (eg. when using non-GitLab managed clusters) but you'll find in general that even if your deployment is working this KUBE_NAMESPACE will not be detected by many other GitLab features (eg. pod terminals, pod logs, deploy boards) and hence they will all still be looking in the wrong namespace.

Since you've raised this issue it's made me aware that we have documentation that is really misleading our users on this topic and it should be changed. Basically I can see 2 references telling people they can configure this CI variable and in fact I think we should stop telling users to do this as it does not work. In fact we should probably explicitly state that users should not do this.

I can see bad references at:

• https://docs.gitlab.com/ee/user/project/clusters/#gitlab-managed-clusters (should just be removed from docs)

  If you are using Auto DevOps, you will need to explicitly provide the KUBE_NAMESPACE deployment variable that will be used by your deployment jobs, otherwise a namespace will be created for you.

• https://docs.gitlab.com/ee/user/project/clusters/#deployment-variables (should be changed to warn users against trying to overwrite this)

  You can overwrite it to use different one if needed, otherwise the KUBE_NAMESPACE variable will receive the default value.

To the extent it helps we are planning on working on https://gitlab.com/gitlab-org/gitlab-ce/issues/59638 soon which I hope will help with your use case.

@danielgruesso @eread we should prioritise these docs changes as this seems to cause frequent confusion and it should be a small change.

mentioned in issue #32003 (closed)

mentioned in issue #32518 (closed)

mentioned in issue #32991 (closed)

mentioned in issue #33518 (closed)

mentioned in issue #33975 (closed)

marked this issue as related to #34048 (closed)

marked this issue as related to #31755 (closed)

mentioned in issue #34483 (closed)

mentioned in issue #35047 (closed)

mentioned in issue #35499 (closed)

mentioned in issue #35993 (closed)

mentioned in issue #36584 (closed)

changed milestone to %12.8

added priority3 label

added [deprecated] Accepting merge requests label

mentioned in merge request !21482 (closed)

I would like to point out that I have a situation where I need to make KUBE_NAMESPACE the same when deploying different environments (from the same project), because each of the environments (which result in different deployments) need to access the same PVC and PV (which is a NFS volume in my case), and since PVC are namespaced objects, I need all of these deployments in the same namespace (as it's not possible for multiple PVCs to be bound to the same PV).

added customer label

Affecting Silver customer with 3 seats: https://gitlab.zendesk.com/agent/tickets/144392 (internal)

mentioned in issue #38054 (closed)

We now have the environment.kubernetes.namespace field in .gitlab-ci.yml to safely customize the namespace used for non-managed clusters. This way, KUBECONFIG and KUBE_NAMESPACE should be correctly set by GitLab within the deployment job

changed milestone to %12.9

added missed:12.8 label

Closing the issue as #31519 (comment 274131273) is the supported way of customizing namespaces per environment.

closed

added groupenvironments label and removed groupconfigure [DEPRECATED] label