Skip to content

WIP: POC/Discovery for group level clusters

Thong Kuah requested to merge group_k8s_cluster into master

What does this MR do?

Validate and note what assumptions we violate. See what is the list of mergeable MRs

Possible MRs

  1. backend Allow project#deployment_platform to choose group level cluster if match environment scope
  2. backend Create cluster on group level
    • Only install helm and nginx-ingress
    • Refactor Clusters::CreateService to take in either group or project as a param.
    • Refactor Clusters::Applications::ScheduleInstallationService to take in either group or project as a param.
    • Significant code re-use for Projects::ClustersController and Projects::Clusters::ApplicationController
  3. frontend Cluster application install vue component with only helm and ingress
  4. frontend backend Show group level clusters on project cluster page
  5. backend Allow Clusters::Platform::Kubernetes to function with group, not just project
  6. Auto Devops domain - Document that users cannot use the same base domain name for 2 different projects (or prefix with project name)

Discard pile

  • Refactor to have multiple project-namespace pairs
  • Possibly re-use Clusters::Platforms::Kubernetes and refactor to handle projects not `project.
    • Work out how to break compat with KubernetesService cleanly
    • Create the project namespace, persist with a database table (cluster_projects)
  • Persist gitlab-managed-apps namespace into database as well
  • Install cluster applications on project level, into project namespace

POC Tasks

  • Create a group-level cluster
  • install helm
  • install ingress
  • Find appropriate group or project level cluster - Environment scope ?
  • destroy group-level cluster
  • Only allow installing of helm and nginx-ingress for a group-level cluster
  • Project, use group-level cluster for this project
  • [-] Install runner into project level namespace for the cluster

Discovery tasks

  • Precedence of Environment Scope for picking which cluster to deploy to
  • database models we'll use (do we use the same tables or new tables?)
    • Can mostly re-use the same tables
    • clusters can be linked to multiple projects via cluster_projects
  • inheritance of env variables down to project levels
    • The variables involved here looks like it will be :
      • KUBE_URL
      • KUBE_TOKEN
      • KUBE_NAMESPACE
      • KUBECONFIG
      • KUBE_CA_PEM
      • KUBE_CA_PEM_FILE
  • Groups, subgroups edge cases
  • k8s namespaces we'll use (different for each project?)
    • Decision Start with using the generated namespace each cluster will generate.
    • Unsure what will happen if the user enters one - disallow that for group-level clusters ?
  • Installing helm applications for groups or keep doing this per project, what are the trade-offs?
    • Decision Helm applications should be installed only for the cluster.
    • Installing helm applications for a group-level cluster as per current should be possible with no change.
    • If per project, will have to install into project unique namespaces instead of fixed namespace gitlab-managed-apps. Cannot "share" applications
  • Will RBAC require more consideration to make this work (more service accounts now?)
    • Does not look like it if we are not changing how we map project environments to namespaces
    • Each project namespace which has Helm Tiller will need service account isolated to that namespace
  • how does our solution extend to instance wide as this is coming up soon (ignoring instance wide on gitlab.com as this brings it's own completely different security challenges)
  • Ingress integration : cluster-wide or per project/namespace ?
    • Decision Cluster-wide but should not use the same base domain with 2 different apps!
    • Consider hiding the help text under the domain name input as it encourages using the same nip.io domain for both. Or we prefix this domain name with the project name or something else to ensure they do not conflict.
  • Prometheus integration : How will prometheus work at the group level? Currently, prometheus is a service-level integration, so there's nothing at the group level. How will we have service integration for prometheus? This leads to other questions such as how to manage it or how to configure custom metrics.
    • Decision 2nd iteration
    • (Dylan) Prometheus also makes most sense at group level but we just need the monitoring team to take a look at what it will take to support monitoring multiple apps (ie. how much code has assumed there is only a single app running?)
  • Runner integration : How will runner work at the group level ?
    • Decision 2nd iteration
    • (Dylan) Runner can be installed at group level, but it just requires a little work. I think group level is a more useful solution here anyway and the need for project level may not be that great
  • Jupyterhub integration : Integration with just one project ?
    • Decision 2nd iteration
    • (Dylan) JupyterHub may also end up making sense to run at the group level too since it should handle multiple users etc quite well. Since it's heavy-weight software it will probably end up being useful multi-tenanted too
  • Auto DevOps domain: currently, every project has a field for "auto devops domain", that doesn't work well when the cluster is at the group level (or multiple clusters at the group level).
    • (Daniel) One possible solution would be to NOT allow project-level domain configuration when there's group level clusters (thought it feels heavy-handed).
  • Who can access group-level clusters? I'd imagine (just as in project-level) that group admin & maintainer are able to manage and developer can view.
    • This should be the same as project-level cluster
  • Auto DevOps
    • See Environment scope
    • $CI_ENVIRONMENT_SLUG which expands to e.g. production. What does it mean to deploy project X into production environment ? How will that clash with deploying project Y into production environment too ?
  • Compatibility with KubernetesService
    • instance level cluster required replacement

What are the relevant issue numbers?

POC for https://gitlab.com/gitlab-org/gitlab-ee/issues/7412

Does this MR meet the acceptance criteria?

Edited by 🤖 GitLab Bot 🤖

Merge request reports