Remove diffence
What does this MR do?
Per https://gitlab.com/gitlab-org/gitlab-ee/issues/10786#note_155671133 exploration the vast majority of the runtime during a secrets
scan is taken up by the diffence
stage`.
By removing diffence we are disabling a large amount of regexes, however I believe we can achieve the same behavior by porting over all additional regexes to the gitleaks blacklist and our rule list.
I'm leaving this as if this is deemed important enough this can be shipped as-is and we can update with the necessary regexes as a followup.WIP
until I can port over all the remaining regexes to prevent regression but
The current expectations already had overlap so removing diffence
only involved updating the scanner and metadata associated with the testdata/id_rsa
key to report from gitleaks instead:
❯ cat test/expect/gl-sast-report.json | jq '.vulnerabilities | map(.cve)'
[
"testdata/main.go:8a22accb113d641b78b389ccce92fca96acbe2fd4f1701ead6783768fdbe9d8a:AWS",
"testdata/main.go:9ac057c201cda016677f2ddeb03d4c59991007a13b9e07917d1f8782e9564970:Generic API Key",
"testdata/main.go:f4ac6d9fd61b258a20a5d9c5aea0e0b48abc60d50edd149021d875066e6c592b:Generic API Key",
"testdata/main.go:72c5f23ed973b0cc312a2c05cae2798ec924f82e3154c86af53e9db0101ec22c:Generic API Key",
"testdata/main.go:ecd55deb344afa69c181ae430c0352de88dd565ab7ed454535824d62f18882fd:Generic API Key",
"testdata/main.go:54767793881cfed5d45a1d28b318679f6f930fc32e84236200f56dcfd98d8b73:Heroku API Key",
"testdata/main.go:430e004686f8f9c2d11ce84da58bd94d1fceb70b0296e46dd8e1ca059ebf7e92:RSA",
"testdata/id_rsa:Private SSH key",
"testdata/main.go:0b16c77410a5918254c1b2bdc7576b87d9ffc7dcedf29b7bdca0b423f6209009:Stripe"
]
~/code/gl/security_products/analyzers/secrets remove-diffence
❯ cat test/fixtures/gl-sast-report.json | jq '.vulnerabilities | map(.cve)'
[
"testdata/main.go:8a22accb113d641b78b389ccce92fca96acbe2fd4f1701ead6783768fdbe9d8a:AWS",
"testdata/main.go:9ac057c201cda016677f2ddeb03d4c59991007a13b9e07917d1f8782e9564970:Generic API Key",
"testdata/main.go:f4ac6d9fd61b258a20a5d9c5aea0e0b48abc60d50edd149021d875066e6c592b:Generic API Key",
"testdata/main.go:72c5f23ed973b0cc312a2c05cae2798ec924f82e3154c86af53e9db0101ec22c:Generic API Key",
"testdata/main.go:ecd55deb344afa69c181ae430c0352de88dd565ab7ed454535824d62f18882fd:Generic API Key",
"testdata/main.go:54767793881cfed5d45a1d28b318679f6f930fc32e84236200f56dcfd98d8b73:Heroku API Key",
"testdata/id_rsa:8bcac7908eb950419537b91e19adc83ce2c9cbfdacf4f81157fdadfec11f7017:RSA",
"testdata/main.go:430e004686f8f9c2d11ce84da58bd94d1fceb70b0296e46dd8e1ca059ebf7e92:RSA",
"testdata/main.go:0b16c77410a5918254c1b2bdc7576b87d9ffc7dcedf29b7bdca0b423f6209009:Stripe"
]
Other than the order change, here's the diff of the same id_rsa
report from gitleaks
instead:
diff --git a/test/expect/gl-sast-report.json b/test/expect/gl-sast-report.json
index 86f46d3..be23d62 100644
--- a/test/expect/gl-sast-report.json
+++ b/test/expect/gl-sast-report.json
@@ -199,15 +199,15 @@
},
{
"category": "sast",
- "name": "SSH private key",
- "message": "SSH private key",
- "description": "SSH private key detected; please remove and revoke it if this is a leak.",
- "cve": "testdata/id_rsa:Private SSH key",
+ "name": "RSA private key",
+ "message": "RSA private key",
+ "description": "RSA private key detected; please remove and revoke it if this is a leak.",
+ "cve": "testdata/id_rsa:8bcac7908eb950419537b91e19adc83ce2c9cbfdacf4f81157fdadfec11f7017:RSA",
"severity": "Critical",
"confidence": "Unknown",
"scanner": {
- "id": "diffence",
- "name": "Diffence"
+ "id": "gitleaks",
+ "name": "Gitleaks"
},
"location": {
"file": "testdata/id_rsa",
@@ -219,9 +219,9 @@
},
"identifiers": [
{
- "type": "diffence_rule_id",
- "name": "Diffence rule ID Private SSH key",
- "value": "Private SSH key"
+ "type": "gitleaks_rule_id",
+ "name": "Gitleaks rule ID RSA",
+ "value": "RSA"
}
]
},
What are the relevant issue numbers?
https://gitlab.com/gitlab-org/gitlab-ee/issues/10786
Does this MR meet the acceptance criteria?
-
Changelog entry added -
Documentation created/updated for GitLab EE, if necessary -
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests added for this feature/bug -
Job definition updated, if necessary -
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer