Support Vault EE Namespaces with JWT via YAML configuration

Problem to solve

GitHub currently supports namespace specifications for Vault Enterprise. GitLab needs to be able to expand the BYOV instance to support Namespaces in Vault

https://github.com/marketplace/actions/vault-secrets#vault-enterprise-features

Proposal

Update the detailed syntax to support new optional namespace parameter, for example:

  secrets:
    SSL_PRIVATE_KEY:
      vault:
        namespace: infrastructure
        engine:
          name: kv-v2
          path: aws
        path: gitlab-test/ssl
        field: private-key

and pass it to Runner to use when authenticating / reading secrets.

Reference documentation: https://www.vaultproject.io/docs/enterprise/namespaces.

Additional notes

Within !80590 (merged), support for Vault EE Namespaces with JWT is already in place via a CI variable.

This issue will implement a YAML configuration to support Vault EE Namespaces with JWT.

added Category:Secrets Management needs weight sectionops + 1 deleted label

cc @krasio

changed milestone to %Backlog

changed the description

changed title from Support Vault EE Names with JWT to Support Vault EE Namespaces with JWT

Setting label(s) ~"devops::release" based on ~"group::release management".

added devopsrelease [DEPRECATED] label

unassigned @jreporter

Looking to add for weighting in %13.7

@jreporter I'll set weight to 2. I suspect we may be able to use namespaces even now but in order to test I need Vault Enterprise. We'll also need it during development.

This is SO exciting 🥳 thanks @krasio

Can you update the proposal with what you will do @krasio ?

@krasio Hashicorp has publicly available enterprise images & binaries for testing purposes like this. You can use them without a license, but the server will seal itself after 30 minutes.

docker run --rm -ti hashicorp/vault-enterprise:latest
Couldn't start vault with IPC_LOCK. Disabling IPC_LOCK, please use --privileged or --cap-add IPC_LOCK
==> Vault server configuration:

             Api Address: http://0.0.0.0:8200
                     Cgo: disabled
         Cluster Address: https://0.0.0.0:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.5.5+ent
             Version Sha: 05fe30b1eeadd105312c529c495f107dc54b40c9

==> Vault server started! Log data will stream in below:

thanks @williams.brian-heb

We are working with our Hashi team too see if there is another way to work around this too @krasio

@williams.brian-heb Thank you!

Can you update the proposal with what you will do @krasio ?

@jreporter Done - #255619.

Thank you

@jreporter @krasio I'd like to be able to specify different namespaces for the authentication and the secret engine. At H-E-B, we have multiple different "platforms" that engineers use to manage and deploy applications. We have a nested namespace structure that allows these platforms to become "delegated administrators" - Meaning that they have a Vault namespace that they own and maintain. So, our namespace structure looks like this:

-- delegated
  |-- platform-a - Has a shared auth mount
     |-- app-1 - Secrets engines
     |-- app-2 - Secrets engines
     |-- app-3 - Secrets engines
  |-- platform-b - Has a shared auth mount
     |-- app-1 - Secrets engines
     |-- app-2 - Secrets engines
     |-- app-3 - Secrets engines

So, with this setup, we'd want the configuration to look something like this:

  secrets:
    SSL_PRIVATE_KEY:
      vault:
        auth:
          namespace: delegated/platform-a
        engine:
          name: kv-v2
          path: aws
          namespace: delegated/platform-a/app-1
        path: gitlab-test/ssl
        field: private-key

This is super helpful @williams.brian-heb

Are there multiple namespaces per job in your case or is there a single namespace per project in this platform structure?

@jreporter Each project has its own namespace for secrets, but there is an auth mount in the parent namespace which is shared between all projects. If we were to get secrets using the vault CLI, the operations would look like this:

export VAULT_FORMAT=json

# Login w/ JWT
vault write -namespace=delegated/platform-a \
  auth/jwt/login \
  role_name=app-1 \
  jwt="$CI_JOB_JWT" | jq -r .auth.client_token | vault login -

vault kv get -namespace=delegated/platform-a/app-1 kv/credentials
# output
{
  "request_id": "72c04402-6cd8-b26a-dfb5-e21da9622a9e",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "secret1": "foo",
      "secret2": "bar"
    },
    "metadata": {
      "created_time": "2020-11-05T15:12:09.0801727Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "warnings": null
}

For the kv engine, it would be uncommon for someone to use more than one namespace post-login. However, other engines like PKI are better being shared between projects, and would be unlikely to be placed inside the same namespace as the kv engine. Does that help?

Gotcha thanks @williams.brian-heb

@williams.brian-heb Thank you, this is very helpful!

Because of the way you can specify the namespace in requests to Vault I was hoping we can just use namespaces even now. If we take your example from above

vault kv get -namespace=delegated/platform-a/app-1 kv/credentials

should work the same as

vault kv get delegated/platform-a/app-1/kv/credentials

In GitLab CI this will look like the following:

secrets:
  CREDENTIALS:
    vault: credentials@delegated/platform-a/app-1/kv/

(we just prefix the engine's path with the namespace).

Unfortunately we can't do the same when authenticating. The path there is like

{namespace}/auth/{auth_path}/login

or in our example

delegated/platform-a/auth/jwt/login

and at the moment we can specify only the auth path.

So if we add new Vault server configuration (e.g. VAULT_AUTH_NAMESPACE) and prefixing the engine path with the namespace we should be able to achieve what you want. I still will have to test this to confirm it will work.

Alternative approach (or next iteration) will be to add support for engine/namespace and auth/namespace in the CI syntax, as you suggested.

Large Enterprise customer https://gitlab.my.salesforce.com/00161000004xJHk is currently building out their secrets management solution and would like the runner to send JWT requests to a different Vault namespace than root.

added customer label

added [deprecated] Accepting merge requests label

added to epic &2868 (closed)

changed milestone to %13.7

mentioned in issue #241756 (closed)

added cicdactive label

added typefeature label

mentioned in issue #273421 (closed)

marked #273421 (closed) as a duplicate of this issue

marked this issue as related to #273421 (closed)

mentioned in issue #28321 (closed)

cc @Kamalakar_Ponaka

changed milestone to %Backlog

@jreporter Is there a reason this got moved to the backlog? This is important for enterprise customers to be able to utilize Vault.

@spouliquen1 we realigned the Release Management team and Secrets Management is now in Configure. @nagyv-gitlab will now take ownership on the timeline for this feature.

@spouliquen1 As my team (myself including) will need some time to get an overview of the existing functionalities and planned developments, I can't promise any deadlines now and don't have a delivery plan yet.

@nagyv-gitlab would you be able to give us an update on the status of this request and when it might be moved from the backlog into an upcoming milestone?

@brian.collins I don't have a timeline yet. As we have some other, high prio and high value features, don't expect this to land before %13.11

changed the description

The way we have implemented is -> Each GitLab group is mapped into Vault namespace and each project repo represents a microservice/application which can be independently deployable to any environment. It's good include group id as well in jwt and have the group id available in jwt/config in vault.

It's good include group id as well in jwt and have the group id available in jwt/config in vault.

@Kamalakar_Ponaka We already have the group id as namespace_id claim - https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/#how-it-works. The reason it's not called group_id is because it can be either group level or user level namespace.

thanks @krasio

Linking the initial ticket to the issue #178164(internal)

added devopsconfigure [DEPRECATED] groupconfigure [DEPRECATED] labels and removed devopsrelease [DEPRECATED] + 1 deleted label

added Alliances label

removed Alliances label

added GitLab Premium label

@jreporter - May i know the timeline of the fix?

Hiya @Kamalakar_Ponaka - since this is a new feature there isn't a fix! But @nagyv-gitlab will be adding a milestone once he prioritizes it for delivery, which may be in the next few months. Thanks for your patience!

@spouliquen1 - We have a very high dependency with this feature on GitLab adoption and has a direct impact on our legacy tools retirement. How can we get this prioritized ?

@Kamalakar_Ponaka Does the workaround described here work for you? #255619 (comment 442689231)

A Premium enterprise customer with 3500+ seats has expressed interest in this. They currently are not aware of a workaround (but we're confirming if #255619 (comment 442689231) is viable).

A note regarding the workaround from a customer:

It works indeed however the structure of the namespace is not what we are looking for. In essence what it does now if we were to place a vault path with the value jwt and a namespace with the value test it will create auth/test/jwt whereas we are interested in having test/auth/jwt which if we understand correctly this feature would allow.

@nagyv-gitlab - With bash script, we can control the namespace before or after since it is our control. What we are specifically looking for is , direct accessing of CI variables from .gitlab-ci.yaml file.

@Kamalakar_Ponaka Sorry, I don't understand this. At what stage of a job's lifecycle would you like to access the CI variables?

variables:
  VAULT_SERVER_URL: "https://hcvault-nonprod.example.com/"
  VAULT_AUTH_PATH: "jwt"
  VAULT_AUTH_ROLE: "myproject-staging"
stages:
  - vault-secrets

vault-secrets:
  stage: vault-secrets
  secrets:
    DATABASE_PASSWORD:
      vault: myproject/staging/db/password@secret
  script:
    # Check job's ref name
    - echo $CI_COMMIT_REF_NAME
    # and is this ref protected
    - echo $CI_COMMIT_REF_PROTECTED
    - echo $VAULT_SERVER_URL
    - echo $DATABASE_PASSWORD
    - cat $DATABASE_PASSWORD    
  tags:
    - linux-docker

@nagyv-gitlab In the above CI file, we don't have a way to specify the namespace information because our JWT Auth engine is in a different namespace.

If the Auth engine is in root namespace, we will not have any problem because we can specify the namespace in KV path but here Auth engine itself is in a different namespace.

Check below where the auth engine is -> it's in the namespace : devops/cicd

If you still have questions, I'm happy to get on a call with you and explain the issue.

Thanks, no more questions about the use case.

A GitLab Premium customer with 13,000 seats is interested in this feature.

• Link to request:https://gitlab.my.salesforce.com/0016100001ANPRt
• Why interested: They are looking for Native integration with an on-prem hosted vault for CI variables (currently we use JWT)
• PM to mention: @jreporter we already talked about this but adding feedback per that issue.

cc @nagyv-gitlab since he is the PM now @shoyle

Vault namespace / masking vault values

@dhershkovitch @nagyv-gitlab I see your comment here and appreciate the update.

Problem to solve: Reduce significant amount of accidental exposure of credentials/tokens in GitLab Job logs.

Customer has asked about this a few times already, but this time another team in their area has expressed interest, so, I'm adding their use case.

A secure coding practices team has also become interested in this feature. They're seeing secrets from vault getting logged in clear text, typically in error logs when things go wrong. They think this feature should help address that concern.

Gitlab CI Job has a way to call Vault and store them as secrets not just Variables but that feature does not support Vault Namespace, so we really can’t use that feature at this time. I believe when we use this feature we will be able to reduce significant amount of accidental exposure of credentials/tokens in Gitlab Job logs.

cc: @jakebert

@shoyle1 @jakebert I think you are interested in #255186 (closed) instead of this issue.

In either case, given the many headcount resets we are short-staffed. I don't know when we might be able to deliver it. I don't have any plans around this issue right now.

Is it possible that we can set this env variable: https://www.vaultproject.io/docs/commands#vault_namespace

i dont know how gitlab is making the communication with vault but if we set the env above we should be able to target our desired namespace, but if gitlab code is not reading this value, then is useless.

GitLab does not recognize the namespace in the variable yet...

That's true @crizstian . All GitLab needs to do is pass the namespace value in header to Vault REST API call. Namespace can be CI variable, CICD project variable or Group variable.

added candidate13.10 label