Skip to content

Merge injected KUBECONFIGs for clusters and agents

Hordur Freyr Yngvasonrequested to merge
merge-injected-kubeconfigs into master

What does this MR do and why?

This makes it easier to migrate one environment at a time from a certificate-based connection to an agent-based connection by

  1. switching just that environment's context
  2. and explicitly setting the default namespace to match the environment's expected namespace

See #335089 (closed)

Screenshots or screen recordings

With the following .gitlab-ci.yml

# .gitlab-ci.yml
test:
  environment: test
  image: 
    name: bitnami/kubectl
    entrypoint: [""]
  script:
    - kubectl config get-contexts
    - kubectl config current-context || echo "No current context"
    - kubectl get pods || echo "Could not get pods"
    - kubectl config use-context cluster-group/agent-config:agent || echo "Context does not exist"
    - kubectl config current-context
    - kubectl --insecure-skip-tls-verify=true get namespaces || echo "Unable to get namespaces"
    - cat $KUBECONFIG
cluster enabled cluster disabled
agent enabled Screenshot_2022-03-05_at_13-24-44_test___1072____Jobs___cluster-group_agent-config Screenshot_2022-03-05_at_13-26-54_test___1074____Jobs___cluster-group_agent-config
agent disabled Screenshot_2022-03-05_at_13-20-17_test___1071____Jobs___cluster-group_agent-config Screenshot_2022-03-05_at_13-27-58_test___1075____Jobs___cluster-group_agent-config

How to set up and validate locally

For a full E2E test like the one above:

Prerequisites

One-time setup steps required to get E2E testing with GitLab CI and a local Kubernetes cluster working:

  1. Install a docker runtime. If you are on macOS, I recommend colima (brew install colima).

  2. Install kubectl (brew install kubectl)

  3. Install kind (brew install kind). We will use this to create the test cluster

  4. Configure a loopback interface for your GDK. Then add kubernetes as an additional binding to the loopback IP in /etc/hosts. For example, if 172.16.123.1 is the IP, then append this to /etc/hosts:

    172.16.123.1 kubernetes

  5. Install https://gitlab.com/hfyngvason/kubectl-gitlab/. This makes it easier to add the cluster as a certificate-based cluster.

  6. In the terminal session you are going to be using, run

    export GITLAB_URL=<YOUR GDK URL>
 export GITLAB_TOKEN=<A PERSONAL API TOKEN FOR YOUR GDK> # the leading space is intentional so it doesn't get saved in your shell history

  7. Verify that kubectl-gitlab is working: Run kubectl gitlab -i ls to see your instance-level clusters. It should return an empty array (unless you have a cluster configured).

  8. Enable HTTPS in your GDK. This is required for the CI/CD tunnel:

    gdk config set https.enabled true
gdk reconfigure
gdk restart

  9. Configure a docker runner in your GDK. Note that this requires gitlab-development-kit!2462 (merged) to function smoothly with HTTPS (currently being merged).

Testing with a cluster

Goal: Reproduce the upper-left quadrant screenshot above with cluster enabled and agent enabled

  1. Create a cluster with kind. Save the following as kind.yaml 
    # kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: 0.0.0.0
    Then run: 
    kind create cluster --config kind.yaml
# change the cluster hostname to kubernetes. This way, we can address the cluster from within a docker container
sed -i -s 's/0\.0\.0\.0/kubernetes/g' "$HOME/.kube/config"
    Then test that the cluster is addressable: 
    kubectl cluster-info
  2. Add the cluster as an instance-level cluster to your local GDK. Important: GITLAB_URL and GITLAB_TOKEN must be set in your environment as described above. 
    kubectl gitlab add -i
  3. Install the GitLab Agent to your cluster:
    1. Create a blank project, add a blank file .gitlab/agents/agent/config.yaml
    2. Navigate to Infrastructure > Kubernetes, click Actions > Connect with an agent, select agent from the dropdown and copy-paste the docker run ... command into your terminal.
  4. We now need to amend the deployment with self-signed certificate for GitLab:
    1. Create a configmap for your self-signed certificate: 
      kubectl -n gitlab-kubernetes-agent create configmap gdk-ca --from-file "localhost.crt"
    2. Edit the deployment: 
      kubectl -n gitlab-kubernetes-agent edit deployment gitlab-agent
      Under containers[0].args (just below a line similar to - wss://gdk.test:3000/-/kubernetes-agent), append 
      - --ca-cert-file=/gdk-ca/localhost.crt
      Under volumeMounts: append: 
      - mountPath: /gdk-ca
  name: gdk-ca
      Under volumes: append: 
      - configMap:
    defaultMode: 420
    name: gdk-ca
  name: gdk-ca
      Now save and close the editor.
  5. Refresh the agents page. The agent should now show as Connected under Connection status in the table.
  6. Add the following .gitlab-ci.yml to the project: 
    test:
  environment: test
  image: 
    name: bitnami/kubectl
    entrypoint: [""]
  script:
    - kubectl config get-contexts
    - kubectl config current-context || echo "No current context"
    - kubectl get pods || echo "Could not get pods"
    - kubectl config use-context cluster-group/agent-config:agent || echo "Context does not exist"
    - kubectl config current-context
    - kubectl --insecure-skip-tls-verify=true get namespaces || echo "Unable to get namespaces"
    - cat $KUBECONFIG
  7. A pipeline should run, and the output should look similar to the screenshot above.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Hordur Freyr Yngvason

Merge request reports