Skip to content

feat: Add ProcessPassthrough functionality

Lucas Charles requested to merge add-processpassthrough into main

What does this MR do?

ProcessPassthrough() provides functionality for generically parsing passthroughs, both file and raw types. This includes sanitization of paths and ensuring files exist.

This code is primarily an extraction from analyzers/semgrep but includes 2 generic changes:

  • dropping filetype suffixing of temp raw files (ioutil.TempFile("", "gosec-config*.json") => ioutil.TempFile("", "rules*"))
  • additional argument for injecting logger interface, provided by implementor (GenericLogger)

Dropping the tempfile naming seems to work fine for semgrep/gosec in testing. Not sure if we'll run into something later with an analyzer that uses some wonky file identification.

Test cases

Raw passthrough

Note the -f arg below for tempfile

❯ go build -o analyzer && GITLAB_FEATURES=sast_custom_rulesets SECURE_LOG_LEVEL=debug ./analyzer run --target-dir test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough

[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ GitLab Semgrep analyzer vnot-configured
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Detecting project
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ Running analyzer
INFO[0000] Loading config from custom ruleset via raw passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:33-07:00] ▶ /usr/local/bin/semgrep -f /var/folders/5k/xrx317y56y50bxqczgqm64cc0000gn/T/rules3823965092 -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ /usr/local/bin/semgrep -f /var/folders/5k/xrx317y56y50bxqczgqm64cc0000gn/T/rules3823965092 -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
ran 1 rules on 3 files: 1 findings
{"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json", "runs": [{"invocations": [{"executionSuccessful": true, "toolExecutionNotifications": []}], "results": [{"locations": [{"physicalLocation": {"artifactLocation": {"uri": "/Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-raw-passthrough/app/app.py", "uriBaseId": "%SRCROOT%"}, "region": {"endColumn": 29, "endLine": 329, "startColumn": 17, "startLine": 329}}}], "message": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "ruleId": "bandit.B506"}], "tool": {"driver": {"name": "semgrep", "rules": [{"defaultConfiguration": {"level": "error"}, "fullDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "id": "bandit.B506", "name": "bandit.B506", "properties": {"precision": "very-high", "tags": ["CWE-502: Deserialization of Untrusted Data", "OWASP-A8: Insecure Deserialization"]}, "shortDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}}], "semanticVersion": "0.55.1"}}}], "version": "2.1.0"}

[INFO] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ Creating report
[DEBU] [Semgrep] [2021-10-07T08:24:34-07:00] ▶ Converting report with the root path:

File passthrough

Note the -f arg below for semgrep_rules

❯ go build -o analyzer && GITLAB_FEATURES=sast_custom_rulesets SECURE_LOG_LEVEL=debug ./analyzer run --target-dir test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ GitLab Semgrep analyzer vnot-configured
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Detecting project
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ Running analyzer
INFO[0000] Using config from custom ruleset passthrough file: semgrep_rules
[DEBU] [Semgrep] [2021-10-07T08:24:27-07:00] ▶ /usr/local/bin/semgrep -f /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep_rules -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
[DEBU] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ /usr/local/bin/semgrep -f /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep_rules -o /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --no-git-ignore --enable-metrics /Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
ran 1 rules on 3 files: 1 findings
{"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json", "runs": [{"invocations": [{"executionSuccessful": true, "toolExecutionNotifications": []}], "results": [{"locations": [{"physicalLocation": {"artifactLocation": {"uri": "/Users/theoretick/code/gl/security-products/analyzers/semgrep/test/fixtures/python-pip-flask-custom-rulesets-with-passthrough/app/app.py", "uriBaseId": "%SRCROOT%"}, "region": {"endColumn": 29, "endLine": 329, "startColumn": 17, "startLine": 329}}}], "message": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "ruleId": "bandit.B506"}], "tool": {"driver": {"name": "semgrep", "rules": [{"defaultConfiguration": {"level": "error"}, "fullDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}, "id": "bandit.B506", "name": "bandit.B506", "properties": {"precision": "very-high", "tags": ["CWE-502: Deserialization of Untrusted Data", "OWASP-A8: Insecure Deserialization"]}, "shortDescription": {"text": "Avoid using `load()`. `PyYAML.load` can create arbitrary Python\nobjects. A malicious actor could exploit this to run arbitrary\ncode. Use `safe_load()` instead.\n"}}], "semanticVersion": "0.55.1"}}}], "version": "2.1.0"}

[INFO] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ Creating report
[DEBU] [Semgrep] [2021-10-07T08:24:28-07:00] ▶ Converting report with the root path:

What are the relevant issue numbers?

Relates to gitlab-org/gitlab#342434 (closed)

Does this MR meet the acceptance criteria?

Edited by Zach Rice

Merge request reports