Add Deploy Tokens to Validity Checks

What does this MR do and why?

When a token is detected by the Secret Detection analyzer, previously, only findings with the token type gitlab_personal_access_token or gitlab_personal_access_token_routable were assigned a status.

This MR updates validity checks so that findings that include a Deploy Token gitlab_deploy_token, are also assigned a status.

Summary of changes

TokenStatusWorker

One of the jobs to the TokenStatusWorker is to build attributes to create FindingTokenStatus for each Finding and then associate the status of the relevant token to each. Rather than look up a token for each Finding, which would result in n+1 queries, all of the FindingTokenStatus attributes are built in a batch, then all of the Tokens are found by their type.

Previously, to match the FindingTokenStatus attributes to tokens, the tokens were stored in a lookup Hash, keyed by the tokens token_digest (token raw value after being sha-ed). This worked because the TokenStatusWorker only dealt with Personal Access Tokens. Deploy tokens do not have a SHA or digest value, but the raw value is encrypted.

To handle this, TokenStatusWorker has been updated so that the tokens lookup Hash is now indexed by raw token value, and the FindingTokenStatus are indexed by raw token, making matching the token status for a FindingTokenStatus easy

TokenLookupService

The TokenLookupService has been updated to search for DeployTokens. To support changes in the TokenStatusWorker, the find function now returns a Hash of tokens, indexed by the raw token value.

Finding

Secret Detection findings have a metadata variable that identified the kind of token that was detected. The JSON looks like:

      "identifiers": [
        {
          "type": "gitleaks_rule_id",
          "name": "Gitleaks rule ID gitlab_personal_access_token",
          "value": "gitlab_personal_access_token"
        }
      ],

Rather than finding the token type in the worker, this MR moves the ability to easily reference this value to the Vulnerabilities::Finding object.

DeployToken

Since this MR enables Deploy Tokens, the Lookup service needed a way to find DeployTokens by their encrypted value. This MR adds the with_encrypted_tokens scope to enable that.

Database Query

This MR adds a new query to find DeployTokens by their encrypted value.

An example query in raw SQL, including 10 fake tokens that have been encrypted, would be:

SELECT
    "deploy_tokens".*
FROM
    "deploy_tokens"
WHERE
    "deploy_tokens"."token_encrypted" IN ('|lgSLH5cdS3ZDgWbchY3jI13oHOVzcfII2PacEmEOOCS0PGOTqRsBLqg=cd34e1504d03', '|FiNVhMTfyIQwTXVb5s7GNs4RPXeEhmXkc2DHzkzxLXCRFtmxj4C7q4c=2b20de0d794c', '|uBzsXO3JjHB1VMJ99+hYOwiTDz06uxTCEwPNonmF08icWUjyPlBNwtM=b4cf922a9150', '|Qp6+PM2rlXXZsGfr2HWlKgWTQp7fh0+OjWl8l74b93q7rG71Qd2t5O4=8884d2bff808', '|Sp0f0qsM681xPBno7d/gNssI0rouwW+ZybvGXawQ54MxnuK3pkU3laI=651e338f6077', '|S7DXI3mqTZFCPX1SIPqkgzlLL+409YfYUuqznTO16OWOEOA3ttyM5FU=663a93f618d3', '|WV4dlbzrRaZAxHrSuUT8346nlTEupKCOCqn5+x9O8ePc1nCOqqbPNTc=e78c5c8123e0', '|KDZupnM0etEG2pRNHtWqFBh11D+oDzEMfTMOchBwtgPcw4/DuSH6n0M=be99e71ea22a', '|WwyYl7+bekZR1kknENkdxvfIMnF9Fnq/n/obx/umcYPtyWBumyWxpI8=080e336e1be2', '|imK12DRnYC+KodeZZr48SkTRHJZHxRUrZyCMjOwnp1WNf+0R7ZAEE1E=a6e204a1d140')

This query can include up to 100 tokens within the IN clause.

PostSQL.ai explain returns 0 rows since it's hard to get a list of Deploy Tokens in production.

References

• Add Secret Detection token status to all other ... (#534788 - closed) • Craig Smith • 18.2 • On track

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. Create a new project

2. In rails console enable the feature flag

   Feature.enable(:validity_checks, new_project_id)

3. Create a deploy token and make note of the token value

4. In the project, add the deploy token value to any of the files, for example the README

5. In the project, enable pipeline secret detection

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

1. Commit the changes so that a pipeline is triggers.

2. Go to the vulnerability report. One vulnerability should be visible.

3. Click the vulnerability to navigate to the finding page. On the finding page, the Validity check should have the value Active secret.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.