Load existing vulnerabilities in batches
What does this MR do and why?
Before this change, we were loading the existing vulnerability records by the project_fingerprint
s of the report findings which was causing really long SQL queries. With this change, we switch to UUID
s and prevent generating long SQL queries by loading the existing records in batches.
Database review
Here is an example query to load vulnerability records
SELECT
"vulnerabilities"."id" AS t0_r0,
"vulnerabilities"."milestone_id" AS t0_r1,
"vulnerabilities"."epic_id" AS t0_r2,
"vulnerabilities"."project_id" AS t0_r3,
"vulnerabilities"."author_id" AS t0_r4,
"vulnerabilities"."updated_by_id" AS t0_r5,
"vulnerabilities"."last_edited_by_id" AS t0_r6,
"vulnerabilities"."start_date" AS t0_r7,
"vulnerabilities"."due_date" AS t0_r8,
"vulnerabilities"."last_edited_at" AS t0_r9,
"vulnerabilities"."created_at" AS t0_r10,
"vulnerabilities"."updated_at" AS t0_r11,
"vulnerabilities"."title" AS t0_r12,
"vulnerabilities"."title_html" AS t0_r13,
"vulnerabilities"."description" AS t0_r14,
"vulnerabilities"."description_html" AS t0_r15,
"vulnerabilities"."start_date_sourcing_milestone_id" AS t0_r16,
"vulnerabilities"."due_date_sourcing_milestone_id" AS t0_r17,
"vulnerabilities"."state" AS t0_r18,
"vulnerabilities"."severity" AS t0_r19,
"vulnerabilities"."severity_overridden" AS t0_r20,
"vulnerabilities"."confidence" AS t0_r21,
"vulnerabilities"."confidence_overridden" AS t0_r22,
"vulnerabilities"."resolved_by_id" AS t0_r23,
"vulnerabilities"."resolved_at" AS t0_r24,
"vulnerabilities"."report_type" AS t0_r25,
"vulnerabilities"."cached_markdown_version" AS t0_r26,
"vulnerabilities"."confirmed_by_id" AS t0_r27,
"vulnerabilities"."confirmed_at" AS t0_r28,
"vulnerabilities"."dismissed_at" AS t0_r29,
"vulnerabilities"."dismissed_by_id" AS t0_r30,
"vulnerabilities"."resolved_on_default_branch" AS t0_r31,
"vulnerabilities"."present_on_default_branch" AS t0_r32,
"vulnerabilities"."detected_at" AS t0_r33,
"findings"."id" AS t1_r0,
"findings"."created_at" AS t1_r1,
"findings"."updated_at" AS t1_r2,
"findings"."severity" AS t1_r3,
"findings"."confidence" AS t1_r4,
"findings"."report_type" AS t1_r5,
"findings"."project_id" AS t1_r6,
"findings"."scanner_id" AS t1_r7,
"findings"."primary_identifier_id" AS t1_r8,
"findings"."project_fingerprint" AS t1_r9,
"findings"."location_fingerprint" AS t1_r10,
"findings"."uuid" AS t1_r11,
"findings"."name" AS t1_r12,
"findings"."metadata_version" AS t1_r13,
"findings"."raw_metadata" AS t1_r14,
"findings"."vulnerability_id" AS t1_r15,
"findings"."details" AS t1_r16,
"findings"."description" AS t1_r17,
"findings"."message" AS t1_r18,
"findings"."solution" AS t1_r19,
"findings"."cve" AS t1_r20,
"findings"."location" AS t1_r21,
"findings"."detection_method" AS t1_r22
FROM
"vulnerabilities"
LEFT OUTER JOIN "vulnerability_occurrences" "findings" ON "findings"."vulnerability_id" = "vulnerabilities"."id"
WHERE
"findings"."uuid" IN ('fbb39e9f-817b-40f7-a654-c8dcac778e64', 'f2e61f74-dd0e-4094-9449-ea2cd22ded74', '8ec164bd-b0db-4ec9-911c-6a8eb6dfb5a6', 'b49fa75d-5be7-437b-9174-5bbe6bdce954', '46e27298-d426-4565-b921-1a3d205c90b0', '472c026b-4725-44e0-b075-cbacd7d6d1dc', 'b33eff15-8090-4d92-a691-7ab4612b6f5e', 'a27ecb09-93d0-4011-b162-9ca2efee7095', '98b5c568-2cc4-4b0e-a57d-6bb73d692248', 'bb370b03-fdcc-4a88-a683-fcb32fafbea7', '045527d2-cc8e-4a74-a97f-cac9ea40e85a', 'c058602f-3c35-4871-a9a2-127610a4b8b6', 'a10d8f60-fe70-4b52-991d-23e16e9942da', '8388b128-31f4-4010-9577-728993b38feb', '9b1fae93-2933-483e-bc2c-e60854426174', '25690de4-c15a-4ae5-a961-18e24a2aa480', '9688a6aa-05be-4905-b9ac-be0951253b41', 'cbd0aa70-500e-4924-915d-86c4df54f2ab', '23cb5977-4fb4-41c4-a7a2-f95e83095c11', 'da32d50c-e34e-4bc1-872e-95dc38ae850b', 'c7d50886-27f7-4f46-81f4-7aa2016e5430', '57fc040f-cfe4-439c-aad0-394e2b1d9065', '20efe5cf-4a02-4e88-b3e2-95428de1e058', 'd3de9a70-2c8b-4b2e-840a-e8f841e25aeb', '8fb7df4d-41ca-4746-b090-a8001df775f2', 'a39e99b4-a92c-4518-97e8-8869fbef5ae1', '9c66a68e-417e-4eef-a64c-f03215fb2533', '91b59f82-4f63-47c3-a4b0-1aa7cb2f296a', '123a0e50-ffb4-4326-b9f2-2e4a01ae38a8', 'c3cca7be-5b33-4402-bf52-2407324c3379', '21778264-7c06-4043-9247-30ed4af7947e', 'b6b21b01-80d5-4a1d-9a06-7d9d641c74bc', 'a12f22b5-6bb7-4d0c-a377-04438a265c06', '9d92b9fd-a26d-43a7-8db5-2955f01147a4', '4263a905-23c6-43c1-bc6b-23a335d738bb', '93de15cc-f898-43f1-9a6f-912d7dd1dd68', '94dcb4eb-9bad-44ef-a593-74228617eb73', '076f43cd-026a-41d6-a8a3-5f903c767f85', '809fa428-10c7-4cf9-885b-d662c4140c1b', '190cf05d-7615-4ac0-8f30-92606e4dbf69', '20b83a88-488b-4ef3-b530-c3cbd54815fe', 'ccc69f9f-b164-48c1-9081-c3e23725013e', 'bee2b65b-168d-4760-b617-771ec3d231cf', '35fc685c-ab2b-4ca0-a1a3-fdb56038fdab', '614ec7ca-631c-4665-afca-4e75766efbb5', '958f73dc-bd87-4480-afd4-9cfc5e7c8d82', '637ec2de-bc87-4f2b-aff1-1b73063c6929', '7c29571f-a836-4cba-acef-5aa2e32a39dc', '18656adb-13df-48bf-bf85-e40e2f34fa23', '59f3f504-e106-4406-b296-10a015437947')
https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/14225/commands/49929
Related to Very large log line and query in StoreSecurityR... (#346581 - closed).
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Mehmet Emin INAC