Add configuration for the KAS CI tunnel feature

Hordur Freyr Yngvason requested to merge add-kas-k8s-proxy-configuration into master

What does this MR do?

KAS has a new endpoint that can proxy calls from GitLab CI to the Kubernetes API server, but it runs on a different port and needs additional configuration to work.

Related issues

gitlab#324272 (closed)


See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion


  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on
  • Pipeline is green on if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • [-] Equivalent MR/issue for the GitLab Chart opened
    • This functionality has been in the chart for a while.

Test plan

  • We are going to set up a self-signed Omnibus accessible from the local machine under the host https://gitlab.test:8443, with KAS on https://gitlab.test:8443/-/kubernetes-agent, and the Kubernetes API proxy on https://gitlab.test:8443/-/kubernetes-agent/k8s-proxy.
  • The goal is to interact with the Kubernets API proxy locally using kubectl.


  1. Add a host alias in /etc/hosts for gitlab.test, pointing to your docker host address. This document uses a loopback alias

  2. Navigate to your Omnibus repository root and check out this branch.

  3. Generate certificates:

    /usr/local/opt/openssl/bin/openssl req -new -x509 \
        -days 365 \
        -newkey rsa:2048 \
        -nodes \
        -subj "/CN=gitlab.test/" \
        -addext "subjectAltName = DNS:gitlab.test" \
        -keyout "gitlab.test.key" \
        -out "gitlab.test.crt"
  4. Run an Omnibus nightly, mounting the certs and relevant cookbooks, and forwarding the container port 443 to the host port 8443:

    docker run \
        -v $(pwd)/gitlab.test.key:/etc/gitlab/ssl/gitlab.test.key \
        -v $(pwd)/gitlab.test.crt:/etc/gitlab/ssl/gitlab.test.crt \
        --privileged \
        -v $(pwd)/files/gitlab-cookbooks/gitlab:/opt/gitlab/embedded/cookbooks/gitlab \
        -v $(pwd)/files/gitlab-cookbooks/gitlab-kas:/opt/gitlab/embedded/cookbooks/gitlab-kas \
        -p 8443:443 \
        --rm -it \
        gitlab/gitlab-ee:nightly bash
  5. Within the omnibus container:

    # Add hosts alias to be locally reachable
    echo ' gitlab.test' >> /etc/hosts
    # Set external URL
    echo 'external_url "https://gitlab.test"' >> /etc/gitlab/gitlab.rb
    echo "letsencrypt['enable'] = false" >> /etc/gitlab/gitlab.rb
    # Enable kas
    echo "gitlab_kas['enable'] = true" >> /etc/gitlab/gitlab.rb
    echo "gitlab_kas['gitlab_ca_certificate_file'] = '/etc/gitlab/ssl/gitlab.test.crt'" >> /etc/gitlab/gitlab.rb
    # Copy self-signed cert before reconfigure (so KAS trusts it)
    cp /etc/gitlab/ssl/gitlab.test.crt /etc/gitlab/trusted-certs/
    # Start runsv. Necessary because we are in a container; see
    /opt/gitlab/embedded/bin/runsvdir-start &
    # Configure and start gitlab
    gitlab-ctl reconfigure
  6. From your local machine, create a cluster using kind with the following configuration:

    # kind-loopback-alias.yaml
    kind: Cluster
    - group:
    version: v1beta2
    kind: ClusterConfiguration
    patch: |
        - op: add
        path: /apiServer/certSANs/-

    Save the above as kind-loopback-alias.yaml and run the following commands:

    # Create cluster
    kind create cluster --config kind-loopback-alias.yaml
    # Use loopback alias for API server address so we can call it from within an agentk container later on
    sed -e s/ -i '' $HOME/.kube/config
  7. From your browser, log into your omnibus instance at https://gitlab.test:8443

    1. Create a project called root/gitlab-agent (arbitrary name),
    2. Add a .gitlab.yml with a dummy job to enable CI:
      # .gitlab-ci.yml
      dummy-job: { script: echo }
    3. Add a file .gitlab/agents/agent/config.yaml with the following content:
      # .gitlab/agents/agent/config.yaml
          level: debug
    4. Create an agent record in GitLab for the above configuration, but do not install it in your cluster. This can also work, but we need extra configuration for the certificates, so for the purpose of testing it is easier to run agentk directly.
      • To create an agent record, navigate to Infrastructure > Kubernetes > GitLab Agent managed clusters and click through
  8. Save the token obtained from the last step into a file gitlab-agent-token in your Omnibus repo root (where you saved gitlab.test.crt).

  9. Locally, run agentk:

    docker run \
        --add-host gitlab.test: \
        -v $(pwd)/gitlab.test.crt:/etc/gitlab.test.crt \
        -v $(pwd)/gitlab-agent-token:/etc/gitlab-agent-token \
        -v $HOME/.kube/config:/etc/kubeconfig \
        -e KUBECONFIG=/etc/kubeconfig \
        --rm \ \
            --kas-address=wss://gitlab.test:8443/-/kubernetes-agent/ \
            --token-file /etc/gitlab-agent-token \
  10. Within the Omnibus container: In a rails console, switch the job status to running to enable that job's token, and grab its kubeconfig

    # gitlab-rails console
    b = Ci::Build.last
    b.update(status: 'running')
  11. On your local machine, copy the kubeconfig YAML from above, and put it in a file $HOME/kas-kubeconfig.yaml. Run

    sed -e s/gitlab.test/gitlab.test:8443/ -i '' $HOME/kas-kubeconfig.yaml
    kubectl --kubeconfig $HOME/kas-kubeconfig --insecure-skip-tls-verify --context root/gitlab-agent:agent cluster-info
Edited by Hordur Freyr Yngvason

Merge request reports