WIP: PoC - provide JWT to CI jobs to be used to authenticate and read secrets from Vault

Updated

This was implemented in !28063 (merged)

What does this MR do?

This MR is a PoC of how we can provide JWT to CI jobs to be used in order to fetch secrets from Vault server. This approach is described under Alternative proposal here - #199737 (comment 282657457).

The authentication is done using Vault's JWT/OIDC Auth Method.

In the JWT we should include enough details so that users can have the flexibility to match it against roles that suit their use case.

How it works

The job will have JWT provided as environment variable named JWT_AUTH. This MR provides it to every job but we can make it configurable per project or even per job. The JWT will have payload as this

        {
          # Issuer
          :iss => "host.docker.internal",
          # Issued at
          :iat => 1583901810,
          # Expiry at
          :exp => 1583901870,
          # Subject (Project Id)
          :sub => "22",
          # Namespace
          :nid => "user:1",
          # User Id
          :uid => "1",
          # Project Id
          :pid => "22",
          # Job Id
          :jid => "1204",
          # Ref name
          :ref => "master",
          # Ref type is one of branch, tag, merge_request
          :ref_type => "branch",
          # Wildcards that protect the ref (empty list if none)
          :protected => ["master"],
          # Environment before expansion (if any)
          :env => "production"
        }    

The expiry time can be made configurable per project. The JWT is encoded using RS256 and signed with a private key. We can generate signing key for each project in which case we'll also need to provide JWKS endpoint for each project. We can use single key per instance (Rails.application.secrets.openid_connect_signing_key) and use the already existing JWKS endpoint (e.g. https://gitlab.com/oauth/discovery/keys). This endpoint is used by Vault to very the token.

The JWT can then be used to authenticate and read secrets from Vault server that is configured to accept it. Interaction with Vault can be done using the CLI or the API (for example with curl).

This approach leaves most of the work to the users but also gives them great flexibility in what they can do, what types of secrets engines to use, etc...

Example

Lets say we have the passwords for our staging and production DB in Vault server that is running on http://192.168.181.123:8200:

vault kv get -field=password secret/gitlab-ci-vault-demo/staging/db
pa$$w0rd

vault kv get -field=password secret/gitlab-ci-vault-demo/production/db
real-pa$$w0rd

We'll need policies that will allow us to read these secrets (one for each secret):

# Read-only permission on 'secret/data/gitlab-ci-vault-demo/production/*' path
path "secret/data/gitlab-ci-vault-demo/production/*" {
  capabilities = [ "read" ]
}

# Read-only permission on 'secret/data/gitlab-ci-vault-demo/staging/*' path
path "secret/data/gitlab/gitlab-ci-vault-demo/staging/*" {
  capabilities = [ "read" ]
}

We'll also need roles that will link the JWT with these policies.

One for staging named auth/jwt/role/gitlab-ci-vault-demo-staging

{
  "role_type": "jwt",
  "policies": ["gitlab-ci-vault-demo-staging"],
  "bound_subject": "5000028",
  "user_claim": "pid"
}

and one for production named auth/jwt/role/gitlab-ci-vault-demo-production

{
  "role_type": "jwt",
  "policies": ["gitlab-ci-vault-demo-production"],
  "bound_subject": "5000028",
  "user_claim": "pid",
  "bound_claims": {
    "ref": "master"
  }
}

We can use bound_claims to specify who is allowed to use the role. In this example we are matching on the ref attribute of the JWT for the production policy. Combined with making master protected branch this gives us control who can read the production secrets.

In order to use the JWT auth method we need to enable and configure it

vault auth enable jwt

vault write auth/jwt/config \
    jwks_url="http://localhost:3001/oauth/discovery/keys" \
    bound_issuer="192.168.181.123"

bound_isuer here means the value specified should match the iss attribute of the JWT.

With the above setup only JWT that has the ref attribute set to master and issuer iss set to 192.168.181.123 can use the gitlab-ci-vault-demo-production role to autheticate. This may seem like a lot of configuration but we'll need it for any of the proposed solutions.

Now we can have job named staging_secrets that will be able to read secrets under secret/gitlab-ci-vault-demo/staging/ but nothing else:

image: vault:1.3.2

staging_secrets:
  script:
    # We can also make this configurable per project in Setting > CI/CD
    # but having it as environment variable gives us more flexibility, e.g.
    # different jobs can use different Vault servers
    - export VAULT_ADDR=http://192.168.181.123:8200
    # Autheticate and get token. Token expiry time and other properties can be configured
    # when configuring JWT Auth - https://www.vaultproject.io/api/auth/jwt/index.html#parameters-1
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=gitlab-ci-vault-demo-staging jwt=$JWT_AUTH)"
    # Now use the VAULT_TOKEN to read the secret and store it in environment variable
    - export PASSWORD="$(vault kv get -field=password secret/gitlab-ci-vault-demo/staging/db)"
    # Use the secret
    - echo $PASSWORD
    # This will fail because the role gitlab-ci-vault-demo-staging can not read secrets from
    # secret/gitlab-ci-vault-demo/production/*
    - export PASSWORD="$(vault kv get -field=password secret/gitlab-ci-vault-demo/production/db)"