GitHub currently supports namespace specifications for Vault Enterprise. GitLab needs to be able to expand the BYOV instance to support Namespaces in Vault
@jreporter I'll set weight to 2. I suspect we may be able to use namespaces even now but in order to test I need Vault Enterprise. We'll also need it during development.
@krasio Hashicorp has publicly available enterprise images & binaries for testing purposes like this. You can use them without a license, but the server will seal itself after 30 minutes.
docker run --rm -ti hashicorp/vault-enterprise:latestCouldn't start vault with IPC_LOCK. Disabling IPC_LOCK, please use --privileged or --cap-add IPC_LOCK==> Vault server configuration: Api Address: http://0.0.0.0:8200 Cgo: disabled Cluster Address: https://0.0.0.0:8201 Go Version: go1.14.7 Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") Log Level: info Mlock: supported: true, enabled: false Recovery Mode: false Storage: inmem Version: Vault v1.5.5+ent Version Sha: 05fe30b1eeadd105312c529c495f107dc54b40c9==> Vault server started! Log data will stream in below:
@jreporter@krasio I'd like to be able to specify different namespaces for the authentication and the secret engine. At H-E-B, we have multiple different "platforms" that engineers use to manage and deploy applications. We have a nested namespace structure that allows these platforms to become "delegated administrators" - Meaning that they have a Vault namespace that they own and maintain. So, our namespace structure looks like this:
-- delegated |-- platform-a - Has a shared auth mount |-- app-1 - Secrets engines |-- app-2 - Secrets engines |-- app-3 - Secrets engines |-- platform-b - Has a shared auth mount |-- app-1 - Secrets engines |-- app-2 - Secrets engines |-- app-3 - Secrets engines
So, with this setup, we'd want the configuration to look something like this:
@jreporter Each project has its own namespace for secrets, but there is an auth mount in the parent namespace which is shared between all projects. If we were to get secrets using the vault CLI, the operations would look like this:
For the kv engine, it would be uncommon for someone to use more than one namespace post-login. However, other engines like PKI are better being shared between projects, and would be unlikely to be placed inside the same namespace as the kv engine. Does that help?
(we just prefix the engine's path with the namespace).
Unfortunately we can't do the same when authenticating. The path there is like
{namespace}/auth/{auth_path}/login
or in our example
delegated/platform-a/auth/jwt/login
and at the moment we can specify only the auth path.
So if we add new Vault server configuration (e.g. VAULT_AUTH_NAMESPACE) and prefixing the engine path with the namespace we should be able to achieve what you want. I still will have to test this to confirm it will work.
Alternative approach (or next iteration) will be to add support for engine/namespace and auth/namespace in the CI syntax, as you suggested.
Large Enterprise customer https://gitlab.my.salesforce.com/00161000004xJHk is currently building out their secrets management solution and would like the runner to send JWT requests to a different Vault namespace than root.
@spouliquen1 we realigned the Release Management team and Secrets Management is now in Configure. @nagyv-gitlab will now take ownership on the timeline for this feature.
@spouliquen1 As my team (myself including) will need some time to get an overview of the existing functionalities and planned developments, I can't promise any deadlines now and don't have a delivery plan yet.
@nagyv-gitlab would you be able to give us an update on the status of this request and when it might be moved from the backlog into an upcoming milestone?
The way we have implemented is -> Each GitLab group is mapped into Vault namespace and each project repo represents a microservice/application which can be independently deployable to any environment. It's good include group id as well in jwt and have the group id available in jwt/config in vault.
Hiya @Kamalakar_Ponaka - since this is a new feature there isn't a fix! But @nagyv-gitlab will be adding a milestone once he prioritizes it for delivery, which may be in the next few months. Thanks for your patience!
@spouliquen1 - We have a very high dependency with this feature on GitLab adoption and has a direct impact on our legacy tools retirement. How can we get this prioritized ?
A Premium enterprise customer with 3500+ seats has expressed interest in this. They currently are not aware of a workaround (but we're confirming if #255619 (comment 442689231) is viable).
It works indeed however the structure of the namespace is not what we are looking for. In essence what it does now if we were to place a vault path with the value jwt and a namespace with the value test it will create auth/test/jwt whereas we are interested in having test/auth/jwt which if we understand correctly this feature would allow.
@nagyv-gitlab - With bash script, we can control the namespace before or after since it is our control. What we are specifically looking for is , direct accessing of CI variables from .gitlab-ci.yaml file.
variables:VAULT_SERVER_URL:"https://hcvault-nonprod.example.com/"VAULT_AUTH_PATH:"jwt"VAULT_AUTH_ROLE:"myproject-staging"stages:-vault-secretsvault-secrets:stage:vault-secretssecrets:DATABASE_PASSWORD:vault:myproject/staging/db/password@secretscript:# Check job's ref name-echo $CI_COMMIT_REF_NAME# and is this ref protected-echo $CI_COMMIT_REF_PROTECTED-echo $VAULT_SERVER_URL-echo $DATABASE_PASSWORD-cat $DATABASE_PASSWORDtags:-linux-docker
@nagyv-gitlab
In the above CI file, we don't have a way to specify the namespace information because our JWT Auth engine is in a different namespace.
If the Auth engine is in root namespace, we will not have any problem because we can specify the namespace in KV path but here Auth engine itself is in a different namespace.
Check below where the auth engine is -> it's in the namespace : devops/cicd
If you still have questions, I'm happy to get on a call with you and explain the issue.
Problem to solve: Reduce significant amount of accidental exposure of credentials/tokens in GitLab Job logs.
Customer has asked about this a few times already, but this time another team in their area has expressed interest, so, I'm adding their use case.
A secure coding practices team has also become interested in this feature. They're seeing secrets from vault getting logged in clear text, typically in error logs when things go wrong. They think this feature should help address that concern.
Gitlab CI Job has a way to call Vault and store them as secrets not just Variables but that feature does not support Vault Namespace, so we really can’t use that feature at this time. I believe when we use this feature we will be able to reduce significant amount of accidental exposure of credentials/tokens in Gitlab Job logs.
In either case, given the many headcount resets we are short-staffed. I don't know when we might be able to deliver it. I don't have any plans around this issue right now.
i dont know how gitlab is making the communication with vault but if we set the env above we should be able to target our desired namespace, but if gitlab code is not reading this value, then is useless.
That's true @crizstian . All GitLab needs to do is pass the namespace value in header to Vault REST API call. Namespace can be CI variable, CICD project variable or Group variable.
Ultimate customer https://gitlab.my.salesforce.com/00161000004yxj9?sdtd=1 with 2500 seats is currently blocked for using Vault integration by this. I've asked them to evaluate the workaround proposed in #255619 (comment 442689231) but they would very much like to see this scoped/ prioritized. Happy to share more info / set up a call for use case details if needed. Thanks!
@nagyv-gitlab Some of our larger customers are anxious for this and there appears to be no clear timeline for availability in an upcoming release. Will this now be worked on as part of 13.11? This is still listed as part of the backlog.
@xyzzy I don't expect any work to happen on this in the next few months. We are very thin in engineering time, and we have other, higher value developments with other or overlapping large customers. At the same time, this issue is on my radar.
The customer he described to me his use cases and those of his colleagues:
We’re using CIFRA/vault for our secrets, and our organization uses enterprises namespaces in vault. Without the support for Enterprise Namespaces, we need to use hacks for what would otherwise be an elegant solution.
However this feature does not work with Enterprise Namespaces in Vault (which CIFRA uses presumably to federate access to different Vaults across our company, for example I am part of namespace: mygroup-dev-)
So instead of that nice, clean, 2 line syntax, we are forced to resort to alternative much more cumbersome, fragile methods / hacks to retrieve our secrets that make writing and maintaining our .gitlab-ci files more complex.
One of my colleagues explored these options. He showcases four different approaches, including the preferred, not working solution in the link sent above. As you can see our options are:
Use a Vault image
This is not a great solution because we use custom .gitlab-ci images elsewhere for other things (helm, awscli, etc.) and would need to bundle the vault-cli with each and every image that needed to retrieve secrets.
Use curl
This is also not an ideal solution as it necessitates sending various headers and tokens and then parsing out your secret from returned JSON, all of which would require much copy + pasting for each pipeline/stage that needed a specific secret, and opening the door to developers forgetting one piece of the curl or another etc. etc.
Use GitLab Secrets Keyword
Unsupported due to our use of namespaces, which is what I'm asking to have our TAM push for on the issue I initially linked to you.
Hide everything in a template
This is our currently solution. We're using a partial template that effectively wraps the curl commands and arguments needed and lets developers consume it by just providing a few variable overrides in their stages.
Thanks @k33g for the clear summary. We plan to provide support once my team will restart focused work secrets management. This is still unplanned though given other priorities.
@nagyv-gitlab A large Premium customer is asking about prioritizing this once other engineering priorities have been addressed. Is there a known timeframe for when we can get back to prioritizing issues like this?
@jwoods06 Unfortunately, I am not able to provide a timeframe. It is too far away in the future.
I am happy to support any community contributions, but we are too thin to work on 4-5 product categories, and - together with product leaders - I think that focusing on other categories provides more value to our users.
@nagyv-gitlab it looks like you're facing a resource constraint here and are focusing on other features deemed more valuable. I'm wondering though, given the large number of customers expressing interest here (some of them very big customers), can the value add of this feature be reassessed?
I am happy to support any community contributions, but we are too thin to work on 4-5 product categories, and - together with product leaders - I think that focusing on other categories provides more value to our users.
I can only emphasize that I'd prioritize it a lot to support a community contribution. I look at this issue almost every month, and assess if we have the bandwidth to plan with it. As we are regularly asked to borrow engineers to other teams, we are really short-stuffed. I am sorry.
Link to request: GitLab team members may click the linked mention to view additional detail for this request within the customer's collaboration project.
Why interested: Customer evaluating the vault integration with GitLab, but identified that namespaces are not supported thus impacting adoption due to limited capabilities of functionality.
Questions: Have we identified a work around for this? Interested on understanding how other customers are going about this.
I don't think this is a workaround for this specific issue; the point of this issue is supporting Vault EE namespace in the GitLab CI yaml syntax, so that the vault CI syntax works. Customers can already (and, have to) have custom script in their jobs to read from Vault, BECAUSE the vault feature does not support a namespace property like is necessary. Having gitlab-runner actually execute the Vault read using only CI syntax is very valuable, and eliminates additional tooling required to be installed on the runner itself. Don't forget, there are shell executors, and Windows executors, not just Docker image based jobs :)
Fair point @brett_jacobson, however the link posted does give you the ability to utilize different namespaces for Vault within a GitLab CI pipeline; I acknowledge it is a departure from the "integrated" approach offered, but please note this particular issue has been open and sitting in the backlog for 18 months with no projected delivery date.
As far as other executors, the example is not dependent on any specific images, in this job, the vault binary is downloaded as part of the script. It is a bit more verbose than the integrated CI but since you're dealing with the vault binary directly it provides full functionality.
There isn't a way to pass a secret between two jobs without spilling it unless you're using cache. That means you're using the same runner for both jobs.
You could also have a runner (or image) that has the vault CLI and all the other stuff that you need. Basically get it down to a single job.
We looked at using an artifacts with a build.env file but the build.env file can be downloaded from the pipeline page.