Skip to content

Update Vuln::Read traversal ids when a namespace changes

What does this MR do and why?

This MR implements sync events attached to the DB level sync events processed to handle effective mirroring of namespace changes to the CI database. This approach was taken to minimise the risk of lost events resulting in desynchronised data. You can read more about this here.

The Project and Namespace Sync events are published as a side effect of the respective ProcessSyncEventsWorker, specifically AFTER the events are already processed for mirroring purposes to minimise latency.

The new Vulnerabilities::ProcessSyncEventsWorker will be subscribed to these events in a follow up MR once deployed, which will allow us to update the traversal ids of vulnerability_read records when these values are updated. The use of each_batch to conduct this update will effectively behave as an early exit for any updates where there are no applicable vulnerability_read records.

DB Queries

Expand
explain SELECT "vulnerability_reads"."vulnerability_id" FROM "vulnerability_reads" WHERE "vulnerability_reads"."project_id" = 278964 ORDER BY "vulnerability_reads"."vulnerability_id" ASC LIMIT 1
Time: 4.729 ms  
  - planning: 4.549 ms  
  - execution: 0.180 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 8 (~64.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/25969/commands/81706

explain SELECT "vulnerability_reads"."vulnerability_id" FROM "vulnerability_reads" WHERE "vulnerability_reads"."project_id" = 278964 AND "vulnerability_reads"."vulnerability_id" >= 3362225 ORDER BY "vulnerability_reads"."vulnerability_id" ASC LIMIT 1 OFFSET 100
Time: 10.290 ms  
  - planning: 3.495 ms  
  - execution: 6.795 ms  
    - I/O read: 6.502 ms  
    - I/O write: 0.000 ms  
Shared buffers:  
  - hits: 7 (~56.00 KiB) from the buffer pool  
  - reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/25969/commands/81705

explain UPDATE "vulnerability_reads" SET "namespace_id" = 9970, "traversal_ids" = '{9970}' WHERE "vulnerability_reads"."project_id" = 278964 AND "vulnerability_reads"."vulnerability_id" >= 3362225 AND "vulnerability_reads"."vulnerability_id" < 3338510
  
Time: 5.012 ms  
  - planning: 3.561 ms  
  - execution: 1.451 ms  
    - I/O read: 1.280 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 6 (~48.00 KiB) from the buffer pool  
  - reads: 1 (~8.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/25969/commands/81704

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Establish a project with vulnerability_read records in a nested namespace (so a namespace that belongs to a parent namespace)
  2. Transfer the sub-namespace that the project belongs to, to another parent level namespace.
  3. Using the rails console, confirm that the vulnerability_read records that belong to the project have been updated such that their traversal_ids field matches that of their owning namespace, which should itself have been updated to match the new parent level namespace.

Related to #437669 (closed)

Edited by Gregory Havenga

Merge request reports