Make variable type for the GitLab CI secret configurable

Release notes

Vault users can easily retrieve secrets from within GitLab CI by using GitLab's Vault integration features. The integration initially created file type secret variables, and this approach had a few shortcomings. To overcome these we are introducing a new, boolean file keyword where you can specify if the returned variable should be stored as a plain environment variable or a file.

For backwards compatibility, the default is to store the variables as file types, that corresponds with setting file:true. At the same time, we recommend using file:false. We will likely change this default in the next major release.

Problem to Solve

We've recently added the secrets: concept to GitLab CI/CD - #218746 (comment 412577121).

Proposal

As it was discussed at the development time and was pointed again with the first feedback, the variable that is created after resolving the secret is of the file type. This requires additional preparation steps in cases, where the variable is supposed to be not file based and simply contain the value.

This was the design choice for the first implementation, but we've discussed the possibility for making this configurable. And it seems that it will be needed 🙂

The proposal here is to:

1. Update the .gitlab-ci.yml syntax to contain a file: true/false configuration setting on the secret level:

   job:
     secrets:
       TEST_SECRET:
         vault:
           engine:
             name: kv-v2
             path: simple-support-test
           path: shared/
           field: KEY_1
         file: false # new field

   or with the condensed format of vault configuration:

   job:
     secrets:
       TEST_SECRET:
          vault:  shared/KEY_1@simple-support-test
          file: false # new field

   As the chosen design is that every secret is turned into variable, the file: setting is added on the same level where vault: or any future secret type is defined. In other words - the file: true/false secret is a setting of the TEST_SECRET secret and not specific to the Vault type.

   For backward compatibility the setting should not be required and either Runner or GitLab should default it to true in that case.

2. Update the Runner API to contain the file entry in the job payload. As above, added as a boolean field file in the secret's hash, at the same level where current vault field exists.

3. Update Runner to handle this new API field and create appropriate variable type.

Further details

We should be sure to keep these secrets as secret and maintain encryption at rest and in transit.