Skip to content

Load existing vulnerabilities in batches

What does this MR do and why?

Before this change, we were loading the existing vulnerability records by the project_fingerprints of the report findings which was causing really long SQL queries. With this change, we switch to UUIDs and prevent generating long SQL queries by loading the existing records in batches.

Database review

Here is an example query to load vulnerability records
SELECT
    "vulnerabilities"."id" AS t0_r0,
    "vulnerabilities"."milestone_id" AS t0_r1,
    "vulnerabilities"."epic_id" AS t0_r2,
    "vulnerabilities"."project_id" AS t0_r3,
    "vulnerabilities"."author_id" AS t0_r4,
    "vulnerabilities"."updated_by_id" AS t0_r5,
    "vulnerabilities"."last_edited_by_id" AS t0_r6,
    "vulnerabilities"."start_date" AS t0_r7,
    "vulnerabilities"."due_date" AS t0_r8,
    "vulnerabilities"."last_edited_at" AS t0_r9,
    "vulnerabilities"."created_at" AS t0_r10,
    "vulnerabilities"."updated_at" AS t0_r11,
    "vulnerabilities"."title" AS t0_r12,
    "vulnerabilities"."title_html" AS t0_r13,
    "vulnerabilities"."description" AS t0_r14,
    "vulnerabilities"."description_html" AS t0_r15,
    "vulnerabilities"."start_date_sourcing_milestone_id" AS t0_r16,
    "vulnerabilities"."due_date_sourcing_milestone_id" AS t0_r17,
    "vulnerabilities"."state" AS t0_r18,
    "vulnerabilities"."severity" AS t0_r19,
    "vulnerabilities"."severity_overridden" AS t0_r20,
    "vulnerabilities"."confidence" AS t0_r21,
    "vulnerabilities"."confidence_overridden" AS t0_r22,
    "vulnerabilities"."resolved_by_id" AS t0_r23,
    "vulnerabilities"."resolved_at" AS t0_r24,
    "vulnerabilities"."report_type" AS t0_r25,
    "vulnerabilities"."cached_markdown_version" AS t0_r26,
    "vulnerabilities"."confirmed_by_id" AS t0_r27,
    "vulnerabilities"."confirmed_at" AS t0_r28,
    "vulnerabilities"."dismissed_at" AS t0_r29,
    "vulnerabilities"."dismissed_by_id" AS t0_r30,
    "vulnerabilities"."resolved_on_default_branch" AS t0_r31,
    "vulnerabilities"."present_on_default_branch" AS t0_r32,
    "vulnerabilities"."detected_at" AS t0_r33,
    "findings"."id" AS t1_r0,
    "findings"."created_at" AS t1_r1,
    "findings"."updated_at" AS t1_r2,
    "findings"."severity" AS t1_r3,
    "findings"."confidence" AS t1_r4,
    "findings"."report_type" AS t1_r5,
    "findings"."project_id" AS t1_r6,
    "findings"."scanner_id" AS t1_r7,
    "findings"."primary_identifier_id" AS t1_r8,
    "findings"."project_fingerprint" AS t1_r9,
    "findings"."location_fingerprint" AS t1_r10,
    "findings"."uuid" AS t1_r11,
    "findings"."name" AS t1_r12,
    "findings"."metadata_version" AS t1_r13,
    "findings"."raw_metadata" AS t1_r14,
    "findings"."vulnerability_id" AS t1_r15,
    "findings"."details" AS t1_r16,
    "findings"."description" AS t1_r17,
    "findings"."message" AS t1_r18,
    "findings"."solution" AS t1_r19,
    "findings"."cve" AS t1_r20,
    "findings"."location" AS t1_r21,
    "findings"."detection_method" AS t1_r22
FROM
    "vulnerabilities"
    LEFT OUTER JOIN "vulnerability_occurrences" "findings" ON "findings"."vulnerability_id" = "vulnerabilities"."id"
WHERE
    "findings"."uuid" IN ('fbb39e9f-817b-40f7-a654-c8dcac778e64', 'f2e61f74-dd0e-4094-9449-ea2cd22ded74', '8ec164bd-b0db-4ec9-911c-6a8eb6dfb5a6', 'b49fa75d-5be7-437b-9174-5bbe6bdce954', '46e27298-d426-4565-b921-1a3d205c90b0', '472c026b-4725-44e0-b075-cbacd7d6d1dc', 'b33eff15-8090-4d92-a691-7ab4612b6f5e', 'a27ecb09-93d0-4011-b162-9ca2efee7095', '98b5c568-2cc4-4b0e-a57d-6bb73d692248', 'bb370b03-fdcc-4a88-a683-fcb32fafbea7', '045527d2-cc8e-4a74-a97f-cac9ea40e85a', 'c058602f-3c35-4871-a9a2-127610a4b8b6', 'a10d8f60-fe70-4b52-991d-23e16e9942da', '8388b128-31f4-4010-9577-728993b38feb', '9b1fae93-2933-483e-bc2c-e60854426174', '25690de4-c15a-4ae5-a961-18e24a2aa480', '9688a6aa-05be-4905-b9ac-be0951253b41', 'cbd0aa70-500e-4924-915d-86c4df54f2ab', '23cb5977-4fb4-41c4-a7a2-f95e83095c11', 'da32d50c-e34e-4bc1-872e-95dc38ae850b', 'c7d50886-27f7-4f46-81f4-7aa2016e5430', '57fc040f-cfe4-439c-aad0-394e2b1d9065', '20efe5cf-4a02-4e88-b3e2-95428de1e058', 'd3de9a70-2c8b-4b2e-840a-e8f841e25aeb', '8fb7df4d-41ca-4746-b090-a8001df775f2', 'a39e99b4-a92c-4518-97e8-8869fbef5ae1', '9c66a68e-417e-4eef-a64c-f03215fb2533', '91b59f82-4f63-47c3-a4b0-1aa7cb2f296a', '123a0e50-ffb4-4326-b9f2-2e4a01ae38a8', 'c3cca7be-5b33-4402-bf52-2407324c3379', '21778264-7c06-4043-9247-30ed4af7947e', 'b6b21b01-80d5-4a1d-9a06-7d9d641c74bc', 'a12f22b5-6bb7-4d0c-a377-04438a265c06', '9d92b9fd-a26d-43a7-8db5-2955f01147a4', '4263a905-23c6-43c1-bc6b-23a335d738bb', '93de15cc-f898-43f1-9a6f-912d7dd1dd68', '94dcb4eb-9bad-44ef-a593-74228617eb73', '076f43cd-026a-41d6-a8a3-5f903c767f85', '809fa428-10c7-4cf9-885b-d662c4140c1b', '190cf05d-7615-4ac0-8f30-92606e4dbf69', '20b83a88-488b-4ef3-b530-c3cbd54815fe', 'ccc69f9f-b164-48c1-9081-c3e23725013e', 'bee2b65b-168d-4760-b617-771ec3d231cf', '35fc685c-ab2b-4ca0-a1a3-fdb56038fdab', '614ec7ca-631c-4665-afca-4e75766efbb5', '958f73dc-bd87-4480-afd4-9cfc5e7c8d82', '637ec2de-bc87-4f2b-aff1-1b73063c6929', '7c29571f-a836-4cba-acef-5aa2e32a39dc', '18656adb-13df-48bf-bf85-e40e2f34fa23', '59f3f504-e106-4406-b296-10a015437947')

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/14225/commands/49929

Related to Very large log line and query in StoreSecurityR... (#346581 - closed).

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mehmet Emin INAC

Merge request reports