GitLab - Kubernetes secrets integration
Release notes
Problem to solve
When I manage my infrastructure, I need to access secrets within the cluster. These secrets should never be available unencrypted outside the cluster.
Use cases
- As an Infrastructure Operator, I need to deploy the project level Runner tokens into the cluster to set up a Runner fleet
- As an Infrastructure Operator, I need to deploy the group level Runner tokens into the cluster to set up a Runner fleet
- As an Application Operator, I need to deploy app-level secrets into the cluster, so I can refer them in my application deployments
Prior art
-
External secrets
❓ - The external secrets operator can connect to various secrets stores, like GCP secrets manager or GitLab project-level environment variables.
- Today, it does not support GitLab group-level environment variables and retrieving the Runner tokens.
- Working with GitLab requires a personal/project access token.
-
SOPS
❌ - A platform agnostic, file-based secrets management tool from Mozilla. It allows encrypting secret files locally, store it in git, and decrypt them where needed.
- We could provide a GitLab side UI where the user inputs the secret, and we create an MR with an encrypted secret.
- We would need to solve the cluster syncing too.
-
Sealed secrets
❓ - A Kubernetes-oriented secrets management tool. The private key lives in the cluster. Thus, only the cluster can decrypt the encrypted secrets.
- interesting discussion on the topic
- We could provide a GitLab side UI where the user inputs the secret, and we create an MR with a sealed secret. So the user does not need access to the Kube API
-
Vault
❌ - Vault is a standalone, open-source secrets management solution
- with sidecar injectors and annotations.
- CSI driver as a mount for secrets in the k8s pod, it’s a GA - beta
- We prefer to not use it, because it would require the management of Vault, injecting the secrets into Vault to allow others to consume those secrets. This is outside the scope for us. Other product categories within GitLab, like Category:Secrets Management might provide deep Vault integrations.
- Cloud secret stores
❌ - We don't want to build a solution around custom cloud secret stores as that would limit usability by users using different clouds.
Proposal
- Extend the agent configuration with the following syntax (two proposals)
- Automatically sync the secrets when they change in GitLab
- Fire an event when the secrets got updated to allow restarting listening applications
❓
manifest_project
Under gitops:
manifest_projects:
- id: my-group/my-project
path: manifests/**/*.yaml
secrets:
runner_token: # deploys the token from my-group/my-project
to:
name: my-project-runner-token
namespace: x_namespace
variables: # deploys variables from my-group/my-project
- name: app-secret
to:
name: cluster-app-secret
namespace: x_namespace
Pros:
- secrets are "managed" together with the manifest configuration
Cons:
- I could not find a syntax to setup group level runner tokens and variables
- Group-level variables would be problematic in terms of access. This might be solved once we support multiple manifest projects and group-level sharing/acceptance.
secrets
Under This approach would define a new, top-level secrets
configuration
secrets:
runner_tokens:
- from: my-group/my-project
to:
name: my-project-runner-token
namespace: x_namespace
variables:
- from: my-group/my-project
name: app-secret
to:
name: cluster-app-secret
namespace: x_namespace
- from: my-group
name: group-secret
to:
name: cluster-group-secret
namespace: x_namespace
Pros:
- It has a natural syntax for group-level secrets
Cons:
- It manages secrets separately from manifests. This could result in a situation where the manifests are removed, but the secrets remain or vice-versa.
- Group-level variables would be problematic in terms of access. This might be solved once we support multiple manifest projects and group-level sharing/acceptance.
Integrate with External secrets
Following the idea of Operator pattern for Kubernetes Agent modules (#350697), we might be able to extend External secrets to better integrate with the agent.
Such integration would remove the requirement to put a fixed access token into the user cluster. The agent would be used for authn&authz. The secrets to sync would be specified as ExternalSecret
kinds provided by the external secrets operator, and the operator would be installed as part of the agentk
installation.
We would need to extend the External secrets operator to support group-level variables and Runner token syncing.
This approach might benefit the users most as they could attach other secret stores using the same technology and the GitOps features of the agent.
Intended users
Feature Usage Metrics
- monthly active secret syncs, when a configured secret needs to be created, updated or removed in the cluster
- MAU number who changes a secret that the agent syncs
Related
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.