Important note: This issue is about interacting with a Vault instance using the GitLab UI; installing a Vault instance for you, or managing secrets that are not used by GitLab CI/CD. For use of GitLab CI/CD with Vault, please see https://gitlab.com/gitlab-org/gitlab-ce/issues/61053.
Description
We manage secrets with project level and group level secret variables (environment variables) as well as service keys at the instance level. There are advantages to using something purpose-built for this, such as Vault, which we plan to bundle with GitLab (omnibus-gitlab#4317 (closed)). Since Vault has a smaller attack surface vs. GitLab, our customer's secrets will be safer here.
This MVC is at its core about adding an interface to that Vault from within GitLab. We should add a new page (perhaps to Operations) that exposes variables from Vault and allows for basic management. Specifically, providing a window into secrets that are stored there that are not otherwise related to GitLab which would otherwise never be surfaced.
The accessor token will need to be determined - it should be specific as possible (probably logged in user?) and not a generic "GitLab" access. https://gitlab.com/gitlab-org/gitlab-ce/issues/61551 will make comparing users between GitLab and Vault easier.
@jdrumtra It's a little early as we're not picking this up any time soon. But we'd love additional clarification on the issue itself of what they'd want to see and why.
I know we would love to have Vault integration as well. It's not that we want to replace the secret management in GitLab, but we'd like all of our CI/CD jobs to start with a Vault token that identifies it as pipeline X so that we can use it to access ephemeral credentials (like AWS tokens) via Vault.
In a simpler form, we'd like to inject some signed identity into the job so we can treat the git repository as an identity (e.g. JWT).
A GitLab CI Vault Authorization method would be a big win!
I would like a Job to gain access to configured Vault policy by the GitLab CI user. This would eliminate the "service/machine" user accounts to the secrets and attach the access granted to GitLab CI the actually human's authorization. I think the GitLab Auth method should also support authentication via Job regardless of the user - the use case would be akin to AWS IAM Service rolls.
There is a GitHub auth method, but no GitLab - auth methods
@jlenny @markpundsack Would you consider opening this up to accepting merge requests? We have a growing list of asks and I would like to intro you to one of them. Standby for invites.
If there are any others following along that might be interested in contributing to this feature please chime in.
@jdrumtra can you please broker a meeting or two with me and customers interested in this feature? East coast US morning/EU time zones preferred if possible, please.
@jlenny I'm a little reluctant for opening complex issues like this, that are difficult to implemented because of complexity, performance and security concerns, for ~"Accepting Merge Requests".
In order to resolve this issue we would also need to refactor some code related to environment variables, and touch internals of GitLab CI. Unless you have someone who is really interested in resolving this, and we know we won't spend months coaching a community merge request, then I can live with opening this for community contributions
I'm working on writing up an issue for injecting unique environment variables per job before a job is spawned (as in not using the existing pre_clone_script, pre_build_script, or post_build_script).
I appreciate #7569 but I'm not after an alternate interface to Vault.
I dont see the value the alt ui would have with without the GitLab Auth Provider mentioned in #note_88053297 where as the GitLab Auth Provider would have significant value with or without the alt ui.
@jlenny Is there any chance we can get this delivered sooner? I have a customer who's been waiting for this for over a year. We have their Executive Business Review coming up soon and it would be amazing to give them news that it would be delivered in Q1.
My thought on how we prioritize this issue (and maybe break it into component parts) is:
I should write a blog post how you can make Vault work with GitLab CI/CD today (manual integration.
We should prioritize the Community Contribution in #53906 (moved) to allow for more direct integration as that community contributor has designed
We should plan for a tighter more bespoke integration with Vault. #53906 (moved) provides a "generic" interface, but that implementation may have limitations relative to how it is done.
We should consider bundling open source Vault into GitLab to allow all users to have first-class integration and secretes management as Vault has "won" that market, and has an open source version
If needed after that, consider what Vault's enterprise customers need on top of that integration.
@brendan +1, I like that plan. You'll learn a lot more about the pain of standing up Vault as we allow more enhanced capabilities and users without Vault become interested and users with Vault ask to get out of the "management of vault" business.
From the customer’s perspective, I think it is important to understand that Vault is essentially two separate things:
a mechanism for providing RBAC for secrets (Authorization based upon MFA, LDAP, User/Pass, Token, etc)
a mechanism for providing encryption at rest that is loosely couple from the storage of secrets (that’s how using a HSM or KMS is used to grant FIPS compliance)
#53906 (moved) provides the first feature, but doesn’t address the second which is key to FIPS. Better documentation or a more robust feature set around Gitlab Secrets can actually provide this just as well.
Drone has a seeming elegant implementation that supports my use case https://docs.drone.io/user-guide/secrets/external/ - I say seemingly because I am just basing this off of the docs and not actually use.
I have a question, with https://gitlab.com/gitlab-org/gitlab-ce/issues/53906 implemented, would that mean that the Vault/aws secrets manager and such integration needs to be done by the user manually? Is that the goal?
I should write a blog post how you can make Vault work with GitLab CI/CD today (manual integration).
If you happened to have written that post, would you mind sharing the URL here?
No worries at all if that didn't happen :)
I'm (we - my colleagues) store a lot of internal secrets for deployment and service accounts in Vault so copying them over to Gitlab secrets would both a pain and have potential for them to get out of sync. We're exploring options how to pull secrets from Vault in Gitlab CI in an efficient way (embedding Vault binary in each build image we deem not efficient due to its size).
Official Gitlab<-->Vault integration (like e.g. in TeamCity) would be a blessing.
Sorry about the replyAll, but there is no other option other than replying
to the incoming address.
You guys do know you are discussing this with users who aren't GitLab
employees - also referencing salesforce URLs (which we can't read). Maybe
a different distribution list?
Yup! One of our company values at GitLab is Transparency, which means everything is made public by default, with only a few exceptions. Because the Salesforce URLs don't give you any private information, it's fine to share publicly.
@liljenstolpe - totally check out our Roadmap for our upcoming features and releases. We’d love to have your voice/contribution in the items that are meaningful to you.
@brendan this didn't make the cut for 11.11 because there were things left in the To-Do list, or because it hasn't been started yet? Wondering about the current actual state.
It is blocked by our Proof of concept issue the precedes it - https://gitlab.com/gitlab-org/gitlab-ee/issues/9981 - will spill into 11.11. The feature freeze for %11.10 is the end of this week, and work hasn't started on that yet.
All, this issue has been overly broad for some time, so I have refined it to be specifically about providing an interface to Vault from within GitLab, for secrets you may be managing. If this is not what you're looking for any longer, your topic may have been moved to one of the other issues which is now part of our broader secrets management category. Please take a look there to determine if what you're looking for is covered in another issue, or please let me know (here is fine) if an important part from your perspective is still missing. Thanks!
Thanks so much. For us, this is exactly what we are looking for. Vault is a great place for us to store secrets (including dynamic ones) because it is FIP 140-2 compliant.
Right now what we do is manually integrate Vault into our CI pipeline using a combination of specialize runner tokens and AWS Auth (we use EC2 over IAM). Because of gitlab-ci.yml includes and a lot of work wrapper our Gitlab Runners, we've been able to abstract out some of the clunkiness. A cleaner simplier way to connect the to would be a big win for us!
I would be happy to share what we do and how we do it (outside a public forum like this).
Hello everyone, I want to draw your attention to https://gitlab.com/gitlab-org/gitlab-ce/issues/61053 where we're finishing up designs for a CI/CD-only integration for secrets from Vault. This isn't the "whole enchilada" as described here, but it is an important piece I know for many of you. It would be great to get your feedback.