Skip to content

Group-level Kubernetes cluster configuration

Problem to solve

Currently, k8s clusters can only be configured at the project level. This can be limiting for teams that want to use the same cluster across all projects/department/company.

Further details

A single cluster configuration that can be leveraged group-wide.

Proposal

Add a new option for "Operations" at the group level that will allow users to access a kubernetes menu item.

The following have been discovered as part of https://gitlab.com/gitlab-org/gitlab-ee/issues/7412:

  1. Only the following group-level applications will be available for MVC:
  • Tiller
  • Ingress
  1. The following applications will be available post-mvc:

  1. Group-level cluster will follow the same licensing approach as project-level clusters: CE will allow for 1 cluster at the group level (or sub-group level) and EE will allow for multiple cluster configuration.

  2. For CE, having a cluster configured at the group level shall not affect the ability for related project to have project-level clusters setup, that is, if Group 1 has a cluster setup, project 1A can have a separate cluster setup (project 1A belonging to Group 1).

  3. Adding a group-level cluster will automatically show that cluster to all children projects

  4. Environment scope for CE will enforce * just as CE project-level cluster configuration does

  5. Project-level cluster trumps group-level cluster, that is, if there's an env scope match, project level always wins.

  6. Sub-groups will determine which cluster to use by looking at closest ancestor with a matching env scope pattern.

  7. Namespace:

    1. Remove the project namespace from group-level clusters completely
    2. Don’t show it in the list when listing clusters, either at the group or project level
    3. Don’t have the field when creating a group-level cluster
    4. Don’t have it when editing a group-level cluster
    5. But keep it when creating/editing a project-level cluster as there’s still reason to keep it
    6. Even though project-level clusters could have custom project namespaces, don't show them in the list of clusters for consistency
    7. The backend still needs to provide KUBE_NAMESPACE to CI/CD pipelines for group-level clusters. I suggest keeping the $PIPELINE_NAME-$PIPELINE_ID format

  8. Managing the cluster can only be done by group maintainers/owners

  9. Removing cluster at the group level will disable the access for its children projects

Discovery issue https://gitlab.com/gitlab-org/gitlab-ee/issues/7412

Follow-up issues:

User stories and mockups

Sidebar and empty state

EE/CE
As an owner or maintainer, I should see a Kubernetes tab in my group sidebar.
As an owner or maintainer, I should see an empty state with a call to action when I have no group clusters.
group__operations--kubernetes-empty-state

Environment scope

EE CE
As an owner or maintainer, I can set the environment scope when creating my group cluster. As an owner or maintainer, I cannot set the environment scope when creating my group cluster. The environment scope is *.
EE-group__operations--kubernetes-create CE-group__operations--kubernetes-create

Applications and domain

EE/CE
As an owner or maintainer, I am able to install Helm Tiller and Ingress on my group level cluster. Note: The only difference between EE/CE mockups would be the ability to set an environment.
As an owner or maintainer, I am able to set a domain for both my group and project level clusters. I am able to set my domain regardless of whether I have installed Helm Tiller and/or Ingress.
Group Project
CE-group__operations--kubernetes-domain-applications-installed CE-project__operations--kubernetes-domain-applications-installed

Improvements: Alternative issues will be made for installing GitLab Runner, Prometheus, and Jupyterhub &114 (closed)

Group clusters

EE CE
As an owner or maintainer, I am able to add multiple group clusters to each group or subgroup. As an owner or maintainer, I can only add one group cluster to each group or subgroup.
EE-group__operations--kubernetes-one-project Add cluster button enabled CE-group__operations--kubernetes-one-project Add cluster button disabled
EE/CE
Group level clusters will be automatically added to projects within that group.

Cluster Overrides

CE
As a user who created a new group cluster, I will be warned if my cluster won't be active for any given project.
CE-group__operations--kubernetes-warning

Improvement: Allow users to specify which projects they would like override. Create new issue

EE CE
As an owner or maintainer, I can have multiple active cluster integrations at various levels on my project. As an owner or maintainer, I can only have one active cluster integration on my project.
EE-project__operations--kubernetes-two CE-project__operations--kubernetes-two
EE-project__operations--kubernetes-four-2 CE-project__operations--kubernetes-four
EE
If multiple clusters exist at the same level, my environment will first look for an exact match, then a partial match, then *
If there are multiple matches, my environment will choose the first match. Project -> Subgroup -> Group
EE-project__operations--kubernetes-four-1
If my environment has no match, it will continue up the chain to find a match. Project -> Subgroup -> Group
CE
As an owner or maintainer, I have the ability to add one project level cluster even if there are group clusters. This will override my group level integration for this project.
CE-project__operations--kubernetes-one-group Add cluster button enabled
CE-project__operations--kubernetes-override-integration

Possible improvement: Explore always keeping the create button enabled. In CE, this would mean owners and maintainers can add as many integrations as they want, but they must select one: either a group or a project level cluster. This removes confusion over why the user cannot create another integration. Create new issue

Editing group clusters

EE/CE
As an owner or maintainer, I can view a group cluster on my project. The cluster links to the group cluster page, and I must make any edits there.
As an owner or maintainer, I cannot remove a group level cluster from a specific project. I can only override the cluster with a project cluster.
CE-project__operations--kubernetes-one-group

Improvement: Allow users to exclude a group level cluster from a specific project. Create new issue

Auto DevOps project settings

EE/CE
As an owner or maintainer, I am no longer able to set a domain within my project Auto DevOps settings
CE-project__settings--ci-cd-autodevops

What does success look like, and how can we measure that?

User no longer copy/paste the same cluster configuration in multiple projects but rather use a group-level cluster

Links / references

/label ~"feature proposal"

Edited by Thong Kuah