Use Hashicorp Vault integration to pass secrets between jobs in a pipeline
I had a short call today with @krasio about the integration, started by initially unrelated question: how we can improve GitLab to pass secrets between jobs in a pipeline without using public and unencrypted artifacts for this.
This addresses the following workflow:
- In my pipeline I have a stage
deploy
with a jobdeploy test
, which deploys my application into a test infrastructure that I own. - In this pipeline I have another stage
test
with jobtest API
, which executes some API tests against the deployed application. - The API requires authentication and the test credentials are created in the
deploy test
step. But they are required intest API
one. - Because of that in the
deploy test
I put the API token into a file that is next saved as an artifact. -
test API
job downloads the artifacts, reads the token from the file and successfully executes the test.
The problem with this workflow is that the artifacts are publicly available. So even if this is a test application, users may leak the credentials to their infrastructure. If the project is public and jobs are public - the credentials become public as well.
And since we plan to integrate with Hashicorp Vault to improve our secrets management and make it more secure, the first idea was to use it also to cover the above problem.
While Phase 1 from the epic fully resolves the problem (since user gets access to his own Vault and can do anything he want), some additional support will be needed when we will go to the Phase 2 and beyond (when we will want to "hide" Vault a little so the script doesn't need to interact directly with it nor Vault doesn't need to be publicly available).
The rough idea for now is that:
- GitLab will be creating some store in the Vault that is dedicated to the pipeline (using the feature of "secrets paths" of Vault). Dedicated means: it lives for the lifetime of the pipeline (but maybe also children/downstream pipelines?) and access is granted only for jobs from this pipeline (and maybe children/downstream?).
- We will create API endpoint in GitLab for adding a variable to this store. Since GitLab already must be accessible by the job (so the sources can be downloaded) we don't need to create any other dependency. For authentication we could use the job token, as we do now for authenticate all operations done in context of the job.
- Secrets from this special store will be passed to the job as variables, just as other secrets defined in project/group/instance level to be taken from Vault.
With this the user will not need to interact with Vault directly, but will only need to know one API endpoint. Also Valut will need to be available only for the GitLab, not for Runner or autoscaled VMs/pods.
Supporting this and the full Vault access from the Phase 1, it should automatically extend also for cases, when GitLab provided Vault would be introduced in the future