Skip to content

Vault requirements for CI and Kubernetes clusters no longer works with version 1.24 and NEW clusters

Details

  • Point of contact for this request: @user
  • If a call is needed, what is the proposed date and time of the call: Date and Time
  • Additional call details (format, type of call): additional details

SRE Support Needed

Problem Statement

During a recent cluster rebuild, version 1.24 is now our primary version of Kubernetes. This version deprecated Service Accounts creating secret objects or tokens for authentication mechanisms by default.

Functional Example

In a GKE v1.23:

$ kubectl create serviceaccount sa-token-demo
serviceaccount/sa-token-demo created
$ k get secrets
NAME                        TYPE                                  DATA   AGE
default-token-hvs55         kubernetes.io/service-account-token   3      66m
sa-token-demo-token-28rqk   kubernetes.io/service-account-token   3      5s

Vs. the new 1.24 cluster:

$ kubectl create serviceaccount sa-token-demo
serviceaccount/sa-token-demo created
$ kubectl get secrets
No resources found in default namespace.

This is problematic as Vault requires some setup that we rely on coming from our service accounts. From our runbook: https://gitlab.com/gitlab-com/runbooks/-/blob/6d5e7441852a4d4d79959d5b425cd9368d695a7e/docs/kube/k8s-cluster-rebuild.md#3d-new-cluster-config-setup and from our Vault documentation: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/vault/administration.md#setting-up-the-external-secrets-operator-for-a-namespace-in-a-kubernetes-cluster

CONTEXT_NAME=$(kubectl config current-context)
KUBERNETES_HOST=$(kubectl config view -o jsonpath='{.clusters[?(@.name == "$CONTEXT_NAME")].cluster.server}')
SA_SECRET=$(kubectl --namespace external-secrets get serviceaccount external-secrets-vault-auth -o jsonpath='{.secrets[0].name}')
SA_TOKEN=$(kubectl --namespace external-secrets get secret ${SA_SECRET} -o jsonpath='{.data.token}' | base64 -d)
CA_CERT=$(kubectl --namespace external-secrets get secret ${SA_SECRET} -o jsonpath='{.data.ca\.crt}' | base64 -d)

vault kv put ci/ops-gitlab-net/config-mgmt/vault-production/kubernetes/CLUSTER host="${KUBERNETES_HOST}" ca_cert="${CA_CERT}" token="${SA_TOKEN}"

However, we are unable to populate SA_SECRET, SA_TOKEN, nor CA_CERT as the secret is never created.

Workaround

We can get around this by manually creating a secret and linking it to the service account. This is precisely what was done in a recent cluster rebuild: production#8083 (comment 1189989936)

  1. Create a secret - we had to do this via a file in order to set the appropriate metadata, there's no secret information in here as Kuberntes creates the token for us:
apiVersion: v1
kind: Secret
metadata:
  name: external-secrets-vault-auth-token-1991j
  namespace: external-secrets
  annotations:
    kubernetes.io/service-account.name: "external-secrets-vault-auth"
type: kubernetes.io/service-account-token
data:
  foo: YmFyCg==

Kubernetes will create the token and crt data objects for us, we're simply left with the data object foo unnecessarily.

  1. And then for the service account, edited manually telling SA where to find the secret associated with it. This was effectively:
  • `kubectl edit sa -n external-secrets external-secrets-vault-auth
  • And adding the below for each of the secrets and SA's we created:
+secrets:
+ - name: external-secrets-vault-auth-token-1991j

This should only be considered temporary as this is not the desired state of service account secrets. Instead, it is desired that secrets are created on an ad-hoc basis, which is why secrets are no longer created by default.

References:

Impact

All NEW clusters created with version 1.24 and later now have a dependency that we need to automate away. Until then, we must rely on a workaround that is currently undocumented, otherwise deployments cannot be completed on a cluster.

Milestones

  • Determine a new or more correct method to apply Kubernetes authentication for Vault
  • Update the documentation for Vault
  • Update our runbook for cluster rebuilds to point to this refreshed documentation
Edited by John Skarbek