Binary Authorization MVC
Problem to solve
Binary Authorization is a deploy-time security control that ensures only trusted container images are deployed on Kubernetes Engine. With Binary Authorization, you can require images to be signed by trusted authorities during the development process and then enforce signature validation when deploying. By enforcing validation, you can gain tighter control over your container environment by ensuring only verified images are integrated into the build-and-release process.
GKE has built-in support for binauthz. The feature is currently in
beta, but will be GA in March/April (before Google Next).
Note that because this feature is dependent on GCP-specific features, it will not be compatible with all k8s clusters/AutoDevOps at MVC.
The basic binary authorization workflow is to have one attestor for the CI system that, if not set, will prevent GKE from deploying output the container image. The way this can work is by using a single attestor (private key) that has the identity of the project or something associated with the CI system as a whole, and not available to users, developers, or anyone else (only owners).
- The binauthz policy in the cluster needs to be set up to only accept images signed with this key - note that third party images are supportable, you can set up a binauthz policy that maps to parts of the registry (i.e., nginx*) We should document this because it will come up.
- Need to identify what is the right key to use (does one exist at the project level already we can use?), or if a new one is needed how it is created/managed
- Using the private key, the CI system attests to the container that is built sending the signature to GKE
- Need to figure out how to enable this without requiring lots of boilerplate code for the CI job. Probably a vendored include of some kind?
- The key should be protected and not visible to the rest of the pipeline, if possible.
- We can modify the AutoDevOps pipeline to detect the presence of special attestation variables, and then automatically add the job if the user add them. This would be a nice way to easily turn on the feature in the pipeline.
- Since GKE already has the binauth feature enabled so it rejects anything not signed by CI
- Preferably through #7840
Multi vs. Single Project Clusters
Because clusters associated with multiple projects and enabling binauthz in a single project does not match, for the MVC we will only support single-project clusters. We can look to add support for group clusters later when we have more time to solve that problem.
We are already signing variables in the gitlab-ci.yml for building signed RPM packages, and there is a method there for providing basic protection of a signing key. The key is only available as environment variables to the signing job, because we use
unset in a
before_script that is only overridden for the signing job:
This is the same or similar approach that anyone signing anything using GitLab is using today.
For a private runner using the shell executor, there is an additional consideration where the gpg certificate store will remain.
gcloud (GCR) credential
Generating the attestation payload and creating the attestation requires gcloud credentials for the container registry. These would need to be stored as environment variables, which is the same thing that anyone pushing to GCR using GitLab CI would need to do.
BinAuthz cannot be used only with Kubectl commands, since configurations need to be done at build time to generate and store attestations. In BinAuthz API, we usually let customer specify which project they will use to store attestations and let them grant the CI/CD pipeline permissions to write. I assume there is already something similar you need to do to get permission to write container images to a registry. See https://medium.com/@gaforres/publishing-google-cloud-container-registry-images-from-gitlab-ci-23c45356ff0e for an example of how this can be setup.
The other thing we will need to solve for is to make sure we're storing the key and credentials in a secure way. This was the bigger concern from the security team. Seems like we can use
attr_encrypted in Rails to do this, but not sure how that decryption works in the runner.
Steps to use binary authorization on GKE
- enable Binary Authorization API in API and Services > Library
- create an attestor in Security > Binary Authorization
- ensure attestor signature is used to sign binaries in pipeline or release
- enable Binary Authorization in the cluster properties (#7840)
Google has a better in depth tutorial here: https://cloud.google.com/binary-authorization/docs/getting-started-cli
The MVC implementation does not use a secure key store or anything like gitlab-ce#53906. It will use protected variables for certificate/keyphrase information as well as gcloud GCR credentials for delivering the attestation. This would be the same as anyone using these today, but we need to be sure as we document this feature that we are clear there is no additional security happening behind the scenes.
We should also document (as in the tutorial) how to set up the GCR credentials to have the absolute minimum needed to create the attestations for the cluster.
There is a dry run mode coming that will not error on deploys but log unsigned ones, which could be interesting to support to test things out.
Once basic CI attestation is setup, you can add more attestors that validate individual things in different ways. For example:
- Manual approvals
- More specific attestations (requirements in yaml, security scans, compliance scans, other activities) One consideration is how does this work with multi-project clusters? It probably doesn't, because they would have to share one policy. A hack would be to share one key.
What does success look like, and how can we measure that?
Number of projects using
binauthz for their deployments.