Provide a kubernetes cluster in GitLab CI

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Close this issue

Release notes

Running tests on top of an ephemeral Kubernetes cluster is a natural requirement for many integration tests, especially in case of a services based architecture. Until now, GitLab users had to set up and tear down these clusters themselves. GitLab now simplifies this process by providing a cluster when the pipeline starts and automatically stopping it when the pipeline is over.

This cluster can be accessed only from the given pipeline, and a Kubernetes context configuration is injected into the CI environment automatically.

Problem to solve

As a developer, I would like to have access to a simple kubernetes cluster without DIND for my GitLab Runner jobs, so I can run my cluster-targeting tests without running real deployments on long-running clusters.

As a developer, I would like to have access to a simple kubernetes cluster throughout a pipeline to run various tests on the cluster at different stages of the pipeline.

Intended users

User experience goal

By injecting a KUBECONFIG, the user can just run kubectl set context ... and then target the cluster with their custom code.

Proposal

Option 1: Use k3s as a service for a job

Document our current solution on docs.gitlab.com and keep it regularly updated (like every quarter).

https://gitlab.com/gitlab-org/cluster-integration/test-utils/k3s-gitlab-ci

Cons

As it creates a k3s service, the cluster is only available until a specific job is running, not throughout the pipeline run.

Pros

It already works, and just needs documentation

Option 2: Run ephemeral clusters for the pipeline

Start a cluster within a cluster (using any of the available options) when the pipeline is created, remove it when the pipeline is over.

In later stages we might want to add monitoring of these clusters to make sure that no cluster got stuck, and we don't have to pay for them extra fees.

Community interest

From: https://kubernetes.slack.com/archives/C09NXKJKA/p1615124689035400

hey guys is there a kubernetes distribution that works as Gitlab CI docker image and creates a working cluster etc.? currently I have to use hacks like KIND or k3s to create Kubernetes for running tests it's painful because when using pure docker-in-docker the kubernetes image won't be cached... derp 20 hours ago

https://github.com/k3s-io/k3s/blob/master/docker-compose.yml
docker-compose.yml

# to run define K3S_TOKEN, K3S_VERSION is optional, eg:
#   K3S_TOKEN=${RANDOM}${RANDOM}${RANDOM} docker-compose up

version: '3'
services:
  server:
    image: "rancher/k3s:${K3S_VERSION:-latest}"
    command: server
    tmpfs:
    - /run
    - /var/run
    privileged: true
    environment:
    - K3S_TOKEN=${K3S_TOKEN:?err}
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - k3s-server:/var/lib/rancher/k3s
    # This is just so that we get the kubeconfig file out
    - .:/output
    ports:
    - 6443:6443
  agent:
    image: "rancher/k3s:${K3S_VERSION:-latest}"
    tmpfs:
    - /run
    - /var/run
    privileged: true
    environment:
    - K3S_URL=https://server:6443
    - K3S_TOKEN=${K3S_TOKEN:?err}
volumes:
  k3s-server: {}

<https://github.com/k3s-io/k3s|k3s-io/k3s>k3s-io/k3s | Added by GitHub

derp  20 hours ago
maybe adapting that would workj?

Chip Zoller  20 hours ago
Look at k3d

derp  20 hours ago
yeah the issue with k3d is that it's not a "docker image"

derp  20 hours ago
it's a script that connects to docker and does stuff

Permissions and Security

Documentation

Availability & Testing

Available Tier

This could be either in GitLab Premium or ~"GitLab Core" depending on the costs it has for gitlab.com.

Actually, we can make it to be in ~"GitLab Core", but have it switched off on gitlab.com by default, and enable it only for GitLab Premium namespaces.

These might come in two iterations, first adding it to GitLab Premium to be SaaS-first, then adding the config options, and move it to ~"GitLab Core"