I would like to have a feature that allows Hashicorp's Vault secrets defined in the global scope. It would be great because with the current setup I have to define secrets in every gitlab ci job even if it's the same secret which is very repetitive. My suggestion is, to make the secrets keyword available under the variable's scope (or it can be a whole new entity outside the job's scope) so that we can access them globally and in the downstream pipelines too.
This issue was automatically tagged with the label grouprunner by TanukiStan, a machine learning classification model, with a probability of 0.93.
If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.
This message was generated automatically.
You're welcome to improve it.
@gitlab-com/pipeline-security-group/backend interested to hear your thoughts on the feasibility of this. The main problem is that when many jobs need the same secret(s), we are not only taking up bandwidth for the secret manager but also significantly slowing down the pipeline.
So the question is: how can we reduce the amount of times we fetch the same secret across the pipeline?
This issue proposes that we somehow make the secret fetching global, and add the secrets to global environment variables.
The problem with this is that it violates the principle of least privilege in that it likely gives the secret to jobs that don't need it
Another older issue proposed creating a secure way to pass the secrets between jobs in a pipeline.
This proposal is very vault dependent, and I'd rather we make a solution that works for all secret managers (including ours). However, the idea is intriguing.
Maybe we make a version of the dotenv report that is secure so that jobs can securely pass environment variables.
Setting aside complexity, what would be the most ideal way to implement something that solves this problem?
My first thought is I'd love to implement the concept of a "profile" in OpenBao, wherein a series of requests occur sequentially on the server and then get combined into a single, unified response. This saves round trips and bandwidth (returning only the needed value(s)) and can occur in the usual "context" of a single auth token. Perhaps with a HCL-based language to match the existing ACL design.
My initial 2c. :-)
However, this wouldn't necessarily initially reduce the requests between separate jobs in the same pipeline. I don't see how the older issue saves bandwidth/requests: you'd still need to talk to Vault for every job if they have separate secrets. I wonder if Runner (in a more privileged, persistent context) could fetch secrets for each job stage and intelligently cache requests a la OpenBao Proxy Request Caching? The cache could be unique to each pipeline and if two jobs both have access to the same secrets, the proxy could verify said access without re-authing and re-fetching from Vault (if Rails indicated it to the Runner out-of-band).
However, this wouldn't necessarily initially reduce the requests between separate jobs in the same pipeline.
@DarrenEastman tagging you here for your thoughts. The number of times we are retrieving the secret is causing issue; wondering if there might be a Runner solution to solve this (or some combination of teams).
If we had the concept of a Global Secret - which seems similar, (except its not Hashicorp Vault), to this solution then maybe there is a path to solving the secrets caching issue. But there are questions:
Will the caching solution work with all runner executors?
Where and how do you store a secret?
For how long?
Do you change the logic in runner to check the cache first for each job with a secret?
However, the more I look at this problem space and the current solution the more I see the logic behind why we went with a job level approach. Theoretically, each CI job in a pipeline, might need a different secret.
This is something I am struggling with as well. I don't think we have a place to store these in GitLab so unless we could make OpenBao's proxy caching work as @cipherboy-gitlab, I'm not sure we meet the security need here.
@shampton we need more help in brainstorming a solution here.
It doesn't seem like we have an easy path forward here. I'd rather not depend on a third party to make this work, so I would prefer not to pursue the OpenBao proxy caching.
What do you all think of option 3 where we make a more secure version of the dotenv report. Some kind of artifact that would be deleted upon pipeline completion that is secure enough to be only read by the runner if granted access in the CI config.
Naturally there are a lot of questions around that, and we'd likely need some design research but it seems like that could work. I'm open to critique though.
@shampton@jocelynjane Am I misreading this issue or is the discussion topic in this thread not the same as the description in the issue?
I reread the description at the top and it really seems more of a DevEx issue than a bandwidth/caching/... issue: it is just repetitive on the part of pipeline authors, if they know they continually need the same sets of secrets, to write out each of those secrets time and time again.
I wonder if secrets can be used in reusable components and if, with better pipeline structuring, they can get better DevEx here? Or is that new functionality that doesn't yet exist?
@cipherboy-gitlab you're right, the issue description is asking for a global configuration for secrets which is more focused on the dev experience of having to write out the same secret for each job. But I think there is an underlying ask in the description for the secrets to also be fetched only once at the beginning of the pipeline and set as variables for the rest of the jobs to use.
My suggestion is, to make the secrets keyword available under the variable's scope (or it can be a whole new entity outside the job's scope) so that we can access them globally and in the downstream pipelines too.
I could be misinterpreting this ask though.
Perhaps there are two things being asked then:
Not having to repeat the same config in each job. They could use the extends keyword but they would still have to add that extend syntax in every job.
Not having every job make the same request for the same secrets. This is the tougher problem.