Add policy violation details, extend finders
What does this MR do and why?
This MR extends finders to be able to lookup security_findings and vulnerabilities by uuid and adds PolicyViolationDetails class to collect information about policy violations for a given merge request. This class will be used to extend the policy bot comment and surface policy violations to the users (done in !146935 (merged)). At the moment, only scan_finding and any_merge_request violations are implemented. license_scanning will be added in a follow-up MR.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Database
FindingsFinder:
findings.with_scan_partition_number.by_uuid(params[:uuids]):
At most 11 uuids are used in the IN statement.
SELECT "security_findings".* FROM "security_findings" INNER JOIN "security_scans" ON "security_scans"."id" = "security_findings"."scan_id" WHERE "security_scans"."status" = 1 AND "security_scans"."project_id" = 55690112 AND "security_scans"."pipeline_id" = 1206454480 AND "security_findings"."partition_number" = "security_scans"."findings_partition_number" AND "security_findings"."uuid" IN ('01433779-d80b-5802-86e5-d2e40aae4e89', '02ae59f7-dd38-569c-83ad-ec248ebe10ab', '02ca1eca-28b9-541c-b0f1-a8d90cbf3d61', '047b8ca4-1708-5d7c-9ff1-f9d01885f8da', '04e7476a-9dd6-5e76-a331-9f4ab0e7704e', '057c5696-e081-57a0-967d-c08ea98f80a2', '0764742e-6b83-52f0-8428-347a1e382c22', 'd20605f2-9ae7-5712-838d-485c558e4e67', '48c909dd-0386-5bda-be5f-d27db882baab', '04afec35-5b47-5119-8663-c042f1f92430', '055c3248-e41f-52ee-9762-e78f43b643b7')Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/27241/commands/84734
Plan cached: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/27241/commands/84735
VulnerabilitiesFinder
vulnerabilities.with_findings_by_uuid(params[:uuids]):
Two queries are performed by Rails - first one fetches vulnerabilities.id which are used in the second one. At most 11 uuids are used in the IN statement.
SELECT DISTINCT "vulnerabilities"."id" FROM "vulnerabilities" LEFT OUTER JOIN "vulnerability_occurrences" "findings" ON "findings"."vulnerability_id" = "vulnerabilities"."id" WHERE "vulnerabilities"."project_id" = 55690112 AND "vulnerabilities"."present_on_default_branch" = TRUE AND "findings"."uuid" IN ('a3a6fab1-c333-5146-8a3f-d4faf9484593', 'a1f95785-6abc-5e79-bc23-0a1142705c27', 'a052ea4d-f53a-503b-935d-08bc4cb1a0d3', 'a0498312-9e46-5ee9-b284-907a130d46ba', 'a036acf9-99f5-5c3a-9ba6-d6782c753860', 'fe751ade-0cc0-5f9e-a522-de20dffdff91', 'fd428700-37a2-5299-9b4d-4b3bdaf686df', 'fcb8adb0-a319-5c64-b50d-6f8eb6dba89a', 'fc7d2e9f-b5c3-51bd-b616-978a555a4415', 'fc27ed72-9a1f-5205-8e25-55fc68bf0a45', 'fc2576fd-dc67-5f8b-8dec-f7a672156286') LIMIT 11Plan: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/27024/commands/84035
SELECT "vulnerabilities"."id" AS t0_r0, "vulnerabilities"."project_id" AS t0_r1, "vulnerabilities"."author_id" AS t0_r2, "vulnerabilities"."created_at" AS t0_r3, "vulnerabilities"."updated_at" AS t0_r4, "vulnerabilities"."state" AS t0_r5, "vulnerabilities"."severity" AS t0_r6, "vulnerabilities"."confidence" AS t0_r7, "vulnerabilities"."severity_overridden" AS t0_r8, "vulnerabilities"."confidence_overridden" AS t0_r9, "vulnerabilities"."title" AS t0_r10, "vulnerabilities"."title_html" AS t0_r11, "vulnerabilities"."description" AS t0_r12, "vulnerabilities"."description_html" AS t0_r13, "vulnerabilities"."report_type" AS t0_r14, "vulnerabilities"."cached_markdown_version" AS t0_r15, "vulnerabilities"."resolved_by_id" AS t0_r16, "vulnerabilities"."resolved_at" AS t0_r17, "vulnerabilities"."confirmed_by_id" AS t0_r18, "vulnerabilities"."confirmed_at" AS t0_r19, "vulnerabilities"."dismissed_at" AS t0_r20, "vulnerabilities"."dismissed_by_id" AS t0_r21, "vulnerabilities"."resolved_on_default_branch" AS t0_r22, "vulnerabilities"."present_on_default_branch" AS t0_r23, "vulnerabilities"."detected_at" AS t0_r24, "vulnerabilities"."finding_id" AS t0_r25, "vulnerabilities"."cvss" AS t0_r26, "findings"."id" AS t1_r0, "findings"."created_at" AS t1_r1, "findings"."updated_at" AS t1_r2, "findings"."severity" AS t1_r3, "findings"."confidence" AS t1_r4, "findings"."report_type" AS t1_r5, "findings"."project_id" AS t1_r6, "findings"."scanner_id" AS t1_r7, "findings"."primary_identifier_id" AS t1_r8, "findings"."project_fingerprint" AS t1_r9, "findings"."location_fingerprint" AS t1_r10, "findings"."name" AS t1_r11, "findings"."metadata_version" AS t1_r12, "findings"."raw_metadata" AS t1_r13, "findings"."vulnerability_id" AS t1_r14, "findings"."details" AS t1_r15, "findings"."description" AS t1_r16, "findings"."solution" AS t1_r17, "findings"."cve" AS t1_r18, "findings"."location" AS t1_r19, "findings"."detection_method" AS t1_r20, "findings"."uuid" AS t1_r21, "findings"."initial_pipeline_id" AS t1_r22, "findings"."latest_pipeline_id" AS t1_r23 FROM "vulnerabilities" LEFT OUTER JOIN "vulnerability_occurrences" "findings" ON "findings"."vulnerability_id" = "vulnerabilities"."id" WHERE "vulnerabilities"."project_id" = 55690112 AND "vulnerabilities"."present_on_default_branch" = TRUE AND "findings"."uuid" IN ('a3a6fab1-c333-5146-8a3f-d4faf9484593', 'a1f95785-6abc-5e79-bc23-0a1142705c27', 'a052ea4d-f53a-503b-935d-08bc4cb1a0d3', 'a0498312-9e46-5ee9-b284-907a130d46ba', 'a036acf9-99f5-5c3a-9ba6-d6782c753860', 'fe751ade-0cc0-5f9e-a522-de20dffdff91', 'fd428700-37a2-5299-9b4d-4b3bdaf686df', 'fcb8adb0-a319-5c64-b50d-6f8eb6dba89a', 'fc7d2e9f-b5c3-51bd-b616-978a555a4415', 'fc27ed72-9a1f-5205-8e25-55fc68bf0a45', 'fc2576fd-dc67-5f8b-8dec-f7a672156286') AND "vulnerabilities"."id" IN (110533839, 110533846, 110533848, 110533851, 110533853, 110533854, 110534338, 110534348, 110534354, 110534356, 110534358)Plan: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/26988/commands/83983
How to set up and validate locally
- In rails console enable the feature flag
Feature.enable(:save_policy_violation_data) - Create a project
- Add CI configuration:
include: - template: Jobs/Secret-Detection.gitlab-ci.yml build-job: script: - echo "Compiling the code..." - echo "Compile complete." - Go to Secure -> Policies and create a new policy:
type: approval_policy name: Test description: '' enabled: true rules: - type: scan_finding scanners: [] vulnerabilities_allowed: 0 severity_levels: [] vulnerability_states: - new_needs_triage - new_dismissed - detected - confirmed - dismissed - resolved branch_type: protected - type: any_merge_request branch_type: protected commits: unsigned actions: - type: require_approval approvals_required: 1 role_approvers: - developer - Create MR adding a new leaked secret. Example:
diff --git a/.env b/.env new file mode 100644 index 0000000000000000000000000000000000000000..ee4bf74ac3b632173dafc09e74ecd68c298bdfa1 --- /dev/null +++ b/.env @@ -0,0 +1 @@ +AWS_TOKEN=AKIAZYONPI3G4JNCCWGQ \ No newline at end of file - After the pipeline completes, verify that jobs pass, artifacts are present and approvals are required
- In rails console, verify the violations:
details = Security::ScanResultPolicies::PolicyViolationDetails.new(MergeRequest.last) details.violations details.new_scan_finding_violations details.previous_scan_finding_violations details.any_merge_request_violations - Merge the secret so that it becomes a previously-existing vulnerability. The vulnerability should show up in
Secure -> Vulnerability report - Create a new MR, adding a new leaked secret. Example:
diff --git a/.env b/.env index ee4bf74ac3b632173dafc09e74ecd68c298bdfa1..d791cd2180779bf094bd017e07b12c1fad506c2c 100644 --- a/.env +++ b/.env @@ -1 +1,2 @@ -AWS_TOKEN=AKIAZYONPI3G4JNCCWGQ \ No newline at end of file +AWS_TOKEN=AKIAZYONPI3G4JNCCWGQ +AWS_TOKEN2=AKIAZYONPI3G4JNCCWGZ \ No newline at end of file - After the pipeline completes, verify the violations again:
details = Security::ScanResultPolicies::PolicyViolationDetails.new(MergeRequest.last) details.violations details.new_scan_finding_violations details.previous_scan_finding_violations details.any_merge_request_violations
Related to #433403 (closed)
Edited by Martin Čavoj