# Proposal Users can currently mark CI variables as "masked" in the UI and they will be hidden from the jobs logs. We will implement the same functionality in step runner: https://gitlab.com/gitlab-org/step-runner/-/issues/39+s. It must be a core part of step runner because we must mask logs at the source. But we can do better than that. Some secrets will be generated during CI and should be marked as "masked" dynamically. And steps should know if they are working with secret data so they can act accordingly. ## Step outputs can be marked as sensitive In the step specification an output can be marked as sensitive. `gen_creds/step.yml`: ```yaml spec: inputs: {} outputs: creds: type: string sensitive: true --- ``` Any value derived from a sensitive value will also be marked sensitive. `.gitlab-ci.yml`: ```yaml my_job: run: - name: gen_creds step: ./gen_creds - name: use_creds step: ./use_creds inputs: wrapped_creds: | # This entire value will be marked sensitive BEGIN ${{ steps.gen_creds.outputs.creds }} END ``` This is [how Terraform handles sensitive variables](https://developer.hashicorp.com/terraform/tutorials/configuration-language/sensitive-variables#reference-sensitive-variables). ## Sensitive values can only be given to sensitive inputs Step inputs can be annotated as sensitive, which allows them to handle sensitive values. If a sensitive value is given to a non-sensitive input it will be an error. This guarantees the step knows which incoming values are sensitive so it can act accordingly. `use_creds/step.yml` ```yaml spec: inputs: wrapped_creds: type: string sensitive: true not_for_creds: type: string --- ``` `.gitlab-ci.yml` ```yaml ... - name: use_creds step: ./use_creds inputs: wrapped_creds: | # This is OK BEGIN ${{ steps.gen_creds.outputs.creds }} END not_for_creds: | # This produces an error BEGIN ${{ steps.gen_creds.outputs.creds }} END ``` ## Masked variables are marked as sensitive All masked CI variables are marked as sensitive when creating the initial step context. `.gitlab-ci.yml`: ```yaml # The variable CREDENTIALS is masked. my_job: run: - name: use_creds step: ./use_creds inputs: wrapped_creds: | # This entire value will be marked sensitive BEGIN ${{ job.CREDENTIALS }} END ``` ## Environment variables are also marked as sensitive Environment variables are also marked as sensitive when they are derived from sensitive values. ```yaml my_job: run: - name: use_creds step: ./use_creds_env env: CREDS: | # This entire variable will be marked sensitive BEGIN ${{ job.CREDENTIALS }} END ``` ## Sensitive values can be exported to a secret file Because steps do not declare their exported variables, the spec cannot be used to mark a variable as sensitive. Instead users should write their export to `secret_file` instead of `export_file`. The effect on the environment will be identical except the value exported will be marked sensitive. `gen_creds/step.yml` ```yaml --- steps: - name: export_creds script: echo CREDS=$(./generate-secret) >> ${{ secrets_file }} # Global environment now has a sensitive value CREDS. - name: export_not_creds script: echo NOT_CREDS=$(./generate-random) >> ${{ export_file }} # Global environment now has a non-sensitive value NOT_CREDS. ``` # Implementation ## Value storage The baseline working directory and global environment are stored in `context.go` in the [`Global` struct](https://gitlab.com/gitlab-org/step-runner/-/blob/43c2d2b3eff5c0dad5170135c996650b53abea28/pkg/context/context.go#L15-16). The rest of the context is stored in the [`Step` struct](https://gitlab.com/gitlab-org/step-runner/-/blob/43c2d2b3eff5c0dad5170135c996650b53abea28/pkg/context/context.go#L46-51). The [function `DigObject`](https://gitlab.com/gitlab-org/step-runner/-/blob/43c2d2b3eff5c0dad5170135c996650b53abea28/pkg/internal/expression/value_type.go#L95-113) is used when evaluating expressions to retrieve values from the step context. All values in the context are stored as `structpb.Value` types. Each value should also store its sensitivity alongside the underlying value. This could be just a struct: ```go type value struct { v structpb.Value sensitive bool } ``` Then `DigObject` can unwrap each value along the path. If any is marked sensitive, the output value will be marked sensitive. ## Output files The `internal/output` package can add another file `secret_file` alongside [the existing `output_file` and `export_file`](https://gitlab.com/gitlab-org/step-runner/-/blob/43c2d2b3eff5c0dad5170135c996650b53abea28/pkg/internal/output/output.go#L29-30). When exporting to the global context it will process `secret_file` the same way as `export_file` but marking all values as sensitive. And when writing outputs to the context they will be matched with their output specification and marked accordingly. ## Step invocation When [adding inputs](https://gitlab.com/gitlab-org/step-runner/-/blob/43c2d2b3eff5c0dad5170135c996650b53abea28/pkg/runner/runner.go#L116) to step invocation parameters, step runner will just check the value sensitive against the spec input sensitivity. ## Dynamic masking Whenever a new sensitive value is generated by an expression, it will be passed to the masking system which will remove it from any subsequent logs. It will be important the step is not invoked until all new masked values are received by the masking system, so a channel rendezvous or lock will be required.