Secret Detection full history scan silently fail and doesn't scan the entire history when GIT_DEPTH is too small
Summary
When running Secret Detection with SECRET_DETECTION_HISTORIC_SCAN: "true"
and SECURE_LOG_LEVEL: "debug"
we can see in the log than only 50 commits have been scanned. Though the corresponding repository has 673 commits for that master
branch:
git rev-list --count master
673
We can see an error
message "object not found" in the [DEBUG] section of the job output (see related issue about mixed log levels).
This is due to GIT_DEPTH
being set to 50
by default, meaning no enough commits are fetched when cloning.
[INFO] [secrets] [2021-04-23T16:36:39Z] ▶ GitLab secrets analyzer v3.19.0
[INFO] [secrets] [2021-04-23T16:36:39Z] ▶ Detecting project
[INFO] [secrets] [2021-04-23T16:36:39Z] ▶ Found project in /builds/gitlab-org/security-products/dependencies/retire.js
[INFO] [secrets] [2021-04-23T16:36:39Z] ▶ Running analyzer
[DEBU] [secrets] [2021-04-23T16:36:39Z] ▶ /builds/gitlab-org/security-products/dependencies/retire.js/.gitlab/secret-detection-ruleset.toml not found, ruleset support will be disabled.
[DEBU] [secrets] [2021-04-23T16:36:40Z] ▶ /usr/local/bin/gitleaks --report /tmp/gitleaks-047666181.json --path /builds/gitlab-org/security-products/dependencies/retire.js --config-path /gitleaks.toml --leaks-exit-code 0
time="2021-04-23T16:36:39Z" level=info msg="opening /builds/gitlab-org/security-products/dependencies/retire.js\n"
time="2021-04-23T16:36:40Z" level=error msg="object not found"
time="2021-04-23T16:36:40Z" level=info msg="scan time: 502 milliseconds 752 microseconds"
time="2021-04-23T16:36:40Z" level=info msg="commits scanned: 50"
time="2021-04-23T16:36:40Z" level=info msg="No leaks found"
[INFO] [secrets] [2021-04-23T16:36:40Z] ▶ Creating report
Steps to reproduce
Example Project
https://gitlab.com/gitlab-org/security-products/dependencies/retire.js/-/jobs/1207686364
What is the current bug behavior?
Secret Detection job silently fails to scan the full history. It only scans the 50 available commits and returns successfully.
No indication is visible unless debug log are enabled.
What is the expected correct behavior?
Secret Detection job fails when trying to scan the full history but some commits are not available due to GIT_DEPTH
value being to small (or any other reason preventing the scan).
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
- Fails the job when a commit can't be scanned.
- Document to proactively set
GIT_DEPTH
when running a full scan history