feat: Add ProcessPassthrough functionality
What does this MR do?
ProcessPassthrough()
provides functionality for generically parsing
passthroughs, both file
and raw
types. This includes sanitization of
paths and ensuring files exist.
This code is primarily an extraction from analyzers/semgrep
but includes 2 generic changes:
- dropping filetype suffixing of temp raw files (
ioutil.TempFile("", "gosec-config*.json")
=>ioutil.TempFile("", "rules*")
) - additional argument for injecting logger interface, provided by implementor (
GenericLogger
)
Dropping the tempfile naming seems to work fine for semgrep/gosec in testing. Not sure if we'll run into something later with an analyzer that uses some wonky file identification.
Test cases
Raw passthrough
Note the -f
arg below for tempfile
❯ go build -o analyzer && GITLAB_FEATURES=sast_custom_rulesets SECURE_LOG_LEVEL=debug ./analyzer run --target-dir test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ GitLab Semgrep analyzer vnot-configured
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Detecting project
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Running analyzer
INFO[0000] Loading config from custom ruleset via raw passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ /usr/local/bin/semgrep -f /var/folders/5k/xrx317y56y50bxqczgqm64cc0000gn/T/rules3823965092 -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ /usr/local/bin/semgrep -f /var/folders/5k/xrx317y56y50bxqczgqm64cc0000gn/T/rules3823965092 -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
ran 1 rules on 3 files: 1 findings
{"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json", "runs": [{"invocations": [{"executionSuccessful": true, "toolExecutionNotifications": []}], "results": [{"locations": [{"physicalLocation": {"artifactLocation": {"uri": "/Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/app/app.py", "uriBaseId": "%SRCROOT%"}, "region": {"endColumn": 29, "endLine": 329, "startColumn": 17, "startLine": 329}}}], "message": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "ruleId": "bandit.B506"}], "tool": {"driver": {"name": "semgrep", "rules": [{"defaultConfiguration": {"level": "error"}, "fullDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "id": "bandit.B506", "name": "bandit.B506", "properties": {"precision": "very-high", "tags": ["CWE-502: Deserialization of Untrusted Data", "OWASP-A8: Insecure Deserialization"]}, "shortDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}}], "semanticVersion": "0.55.1"}}}], "version": "2.1.0"}
[INFO] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ Creating report
[DEBU] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ Converting report with the root path:
File passthrough
Note the -f
arg below for semgrep_rules
❯ go build -o analyzer && GITLAB_FEATURES=sast_custom_rulesets SECURE_LOG_LEVEL=debug ./analyzer run --target-dir test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ GitLab Semgrep analyzer vnot-configured
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Detecting project
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Running analyzer
INFO[0000] Using config from custom ruleset passthrough file: semgrep_rules
[DEBU] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ /usr/local/bin/semgrep -f /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep_rules -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ /usr/local/bin/semgrep -f /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep_rules -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
ran 1 rules on 3 files: 1 findings
{"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json", "runs": [{"invocations": [{"executionSuccessful": true, "toolExecutionNotifications": []}], "results": [{"locations": [{"physicalLocation": {"artifactLocation": {"uri": "/Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/app/app.py", "uriBaseId": "%SRCROOT%"}, "region": {"endColumn": 29, "endLine": 329, "startColumn": 17, "startLine": 329}}}], "message": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "ruleId": "bandit.B506"}], "tool": {"driver": {"name": "semgrep", "rules": [{"defaultConfiguration": {"level": "error"}, "fullDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "id": "bandit.B506", "name": "bandit.B506", "properties": {"precision": "very-high", "tags": ["CWE-502: Deserialization of Untrusted Data", "OWASP-A8: Insecure Deserialization"]}, "shortDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}}], "semanticVersion": "0.55.1"}}}], "version": "2.1.0"}
[INFO] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ Creating report
[DEBU] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ Converting report with the root path:
What are the relevant issue numbers?
Relates to gitlab-org/gitlab#342434 (closed)
Does this MR meet the acceptance criteria?
-
Changelog entry added -
Documentation created/updated for GitLab EE, if necessary -
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests added for this feature/bug -
Job definition updated, if necessary -
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer
Edited by Zach Rice