Group-level Kubernetes cluster configuration
Problem to solve
Currently, k8s clusters can only be configured at the project level. This can be limiting for teams that want to use the same cluster across all projects/department/company.
Further details
A single cluster configuration that can be leveraged group-wide.
Proposal
Add a new option for "Operations" at the group level that will allow users to access a kubernetes menu item.
The following have been discovered as part of https://gitlab.com/gitlab-org/gitlab-ee/issues/7412:
- Only the following group-level applications will be available for MVC:
- Tiller
- Ingress
- The following applications will be available post-mvc:
- Runner https://gitlab.com/gitlab-org/gitlab-ce/issues/51988
- Prometheus https://gitlab.com/gitlab-org/gitlab-ce/issues/51963
- JupyterHub https://gitlab.com/gitlab-org/gitlab-ce/issues/51989
-
Group-level cluster will follow the same licensing approach as project-level clusters: CE will allow for 1 cluster at the group level (or sub-group level) and EE will allow for multiple cluster configuration.
-
For CE, having a cluster configured at the group level shall not affect the ability for related project to have project-level clusters setup, that is, if Group 1 has a cluster setup, project 1A can have a separate cluster setup (project 1A belonging to Group 1).
-
Adding a group-level cluster will automatically show that cluster to all children projects
-
Environment scope for CE will enforce
*
just as CE project-level cluster configuration does -
Project-level cluster trumps group-level cluster, that is, if there's an env scope match, project level always wins.
-
Sub-groups will determine which cluster to use by looking at closest ancestor with a matching env scope pattern.
-
Namespace:
- Remove the project namespace from group-level clusters completely
- Don’t show it in the list when listing clusters, either at the group or project level
- Don’t have the field when creating a group-level cluster
- Don’t have it when editing a group-level cluster
- But keep it when creating/editing a project-level cluster as there’s still reason to keep it
- Even though project-level clusters could have custom project namespaces, don't show them in the list of clusters for consistency
- The backend still needs to provide
KUBE_NAMESPACE
to CI/CD pipelines for group-level clusters. I suggest keeping the$PIPELINE_NAME-$PIPELINE_ID
format
-
Managing the cluster can only be done by group maintainers/owners
-
Removing cluster at the group level will disable the access for its children projects
Discovery issue https://gitlab.com/gitlab-org/gitlab-ee/issues/7412
Follow-up issues:
- Auto DevOps domain will be moved away from project configuration to cluster configuration, to be shown (possibly) along ingress information https://gitlab.com/gitlab-org/gitlab-ce/issues/52363
- Support installing GitLab runner on group level cluster https://gitlab.com/gitlab-org/gitlab-ce/issues/51988
- Show errors if project namespace/SA is missing https://gitlab.com/gitlab-org/gitlab-ce/issues/54506
User stories and mockups
Sidebar and empty state
EE/CE |
---|
As an owner or maintainer, I should see a Kubernetes tab in my group sidebar. |
As an owner or maintainer, I should see an empty state with a call to action when I have no group clusters. |
![]() |
Environment scope
Applications and domain
EE/CE |
---|
As an owner or maintainer, I am able to install Helm Tiller and Ingress on my group level cluster. Note: The only difference between EE/CE mockups would be the ability to set an environment. |
As an owner or maintainer, I am able to set a domain for both my group and project level clusters. I am able to set my domain regardless of whether I have installed Helm Tiller and/or Ingress. |
Group | Project |
---|---|
![]() |
![]() |
Improvements: Alternative issues will be made for installing GitLab Runner, Prometheus, and Jupyterhub &114 (closed)
Group clusters
EE/CE |
---|
Group level clusters will be automatically added to projects within that group. |
Cluster Overrides
CE |
---|
As a user who created a new group cluster, I will be warned if my cluster won't be active for any given project. |
![]() |
Improvement: Allow users to specify which projects they would like override. Create new issue
Possible improvement: Explore always keeping the create button enabled. In CE, this would mean owners and maintainers can add as many integrations as they want, but they must select one: either a group or a project level cluster. This removes confusion over why the user cannot create another integration. Create new issue
Editing group clusters
Improvement: Allow users to exclude a group level cluster from a specific project. Create new issue
Auto DevOps project settings
EE/CE |
---|
As an owner or maintainer, I am no longer able to set a domain within my project Auto DevOps settings |
![]() |
What does success look like, and how can we measure that?
User no longer copy/paste the same cluster configuration in multiple projects but rather use a group-level cluster
Links / references
/label ~"feature proposal"