Automatically detect leaked secrets in CI job logs
Problem to solve
Occasionally, CI users may accidentally print out sensitive information in CI job logs. This might be things like AWS tokens, database passwords, GPG keys, etc.
When these tokens are printed out, there's no way of knowing about it without a manual review of the job log. This is an unlikely thing to do though because no one is expecting to accidentally print sensitive information. If this occurs and isn't caught, the job log is still in it's default state - viewable to whoever it is normally viewable to, even if the group who can view the job log isn't the same as the group who should have access to the secrets that were printed.
Intended users
User experience goal
Upon accidental disclosure of a secret, automatically detect it and mark the job log as private. Also notify the author (and potentially admins?) via a TODO.
Proposal
Many tokens have known patterns. For example AWS access keys are all-caps alpha-numeric and start with either AKIA
or ASIA
. Slack user tokens start with xoxo-
. PGP Private keys contain a block with the text of BEGIN PGP PRIVATE KEY BLOCK
at the start. While not all key / secret types have such easy to recognize identifiers (none of the GitLab ones do to my knowledge) some do and they would be easy to detect.
There is already a feature that parses the job log and uses a regexp to search for the code coverage https://gitlab.com/gitlab-org/gitlab/-/blob/e94bdcb7584bfd7d744c6d7d86cec5d00796793f/lib/gitlab/ci/trace/stream.rb#L76-99 We could extend this to look for the known patterns, and add a configuration area in the CI section to allow the user to specify additional ones.
When the job log is processed, if it gets a hit for any of the known token patterns, the job log visibility should get set to the most restricted possible option and todos should be created for the job author (and possibly project owners or others?).
Phased feature proposals
- Use the default secret file to scan, mark job log as private, todo for user - GitLab Free GitLab Premium
- Use secret customizations, mark job log as private, todo for user - GitLab Free GitLab Premium
- Add to Pipeline Security Tab, Security Dashboard - GitLab Ultimate