Skip to content

WIP: Proof-of-Concept: Add native support in Runner for HashiCorp Vault integration with CI_JOB_JWT

Tomasz Maczukin requested to merge poc/native-support-for-vault-with-jwt into master

What does this MR do?

Implements "native" support for handling HashiCorp Vault secrets in Runner, using the upcoming CI_JOB_JWT change () - in a way described at gitlab#212252 (comment 317690659).

The only difference is that the current implementation relays on a special variable (VAULT_DEFINITION), since support for .gitlab-ci.yml syntax must be added to GitLab in the Rails app. But the syntax of YAML in this variable is exactly what should be added to .gitlab-ci.yml "natively" in the future, and it uses the "target" API structures. The disadvantage of the variable configuration is that we can't overwrite the configuration with extends: and the common parts must be repeated.

Example configuration syntax (taken from a test project at https://gitlab.com/tmaczukin-test-projects/test-vault-jwt):

define secrets job:
  stage: start
  variables:
    VAULT_DEFINITION_ALPHA: |
      my-vault-server:
        server:
          url: https://vault.maczukin.dev
          auth:
            name: jwt
            path: ${CI_SERVER_HOST}/jwt
            data:
              jwt: ${CI_JOB_JWT}
              role: pipeline-job
        secrets:
          SHARED_SECRET:
            engine:
              name: kv2
              path: ${V_SECRETS_PATH}
            path: shared
            # strategy: read # not defining it, since it's default when none action is defined
          WRITE_TEST:
            engine:
              name: kv2
              path: ${V_SECRETS_PATH}
            path: ${V_PIPELINE_PATH}/test
            strategy: write
  script:
  - find ${SHARED_SECRET}
  - randomKey=$(dd if=/dev/urandom count=10 bs=1 | base64 | sha256sum | cut -f 1 -d " ")
  - randomValue=$(dd if=/dev/urandom count=10 bs=1 | base64 | sha256sum | cut -f 1 -d " ")
  - echo "${randomValue:0:10}" > ${WRITE_TEST}/${randomKey:0:10}
  - find ${WRITE_TEST}

Why was this MR needed?

As an experiment on how hard it would be to add such "native" integration using the JWT authentication method. Thanks to this, the users don't need to use the vault image or install vault binary in their job environment nor to execute manually Vault CLI commands.

This is also the first step to make GitLab Runner project to use Vault for storing and sharing secrets between our projects (like credentials for our S3/GCS buckets where we release different binaries).

Are there points in the code the reviewer needs to double check?

Many, starting with missing tests. But this is PoC, and the tests will be added before removing WIP status 🙂

Does this MR meet the acceptance criteria?

  • Documentation created/updated
  • Added tests for this feature/bug
  • In case of conflicts with master - branch was rebased

What are the relevant issue numbers?

Edited by Tomasz Maczukin

Merge request reports