Migration updated location fingerprint for CS findings is wrong
There is a problem with background migration that is merged last week, basically I forgot to include include ::ShaAttribute so it updated location_fingerprint column without proper serialisation for container scanning findings.
Who is effected
Gitlab.com users whose using container scanning and rerun default branch. Self hosted customers won't be effected by this
How many records updated
660_369 has wrong location_fingerprint
How many records are there are the moment
883_342
How problem manifests itself
If Gitlab.com use re runs pipeline for default branch . This will create new vulnerabilities and findings, they are same vulnerabilities but since location_fingerprint data is corrupt it will be perceived as new vulnerabilities by StoreReporService and it will be saved . In other words some users might see spike in their Security Dashboard when they look container scanning vulnerabilities.
How many records are we duplicating at the moment
Since 1st of September (deployment of first MR) we have created,
gitlabhq_production=> SELECT count (id) FROM "vulnerability_occurrences" WHERE "vulnerability_occurrences"."report_type" = 2 AND (created_at > '2020-09-01 14:12:50.453892') ;
count
-------
48220
(1 row)48_220 Container Scanning findings, considering all of them are duplicates (estimation)
23_893 duplicates (information from latest replica db)
How many projects effected
285 (estimation)
240 (information from latest replica db)
What is the fix
!41756 (merged) this fixes background migration, however, we need to create another background migration for removing duplicate findings and vulnerabilities. (first part is completed)
Why two MR is reverted related with this migration
1.) First MR that creates the problem
2.) Second MR that is merged yesterday but not ran
after following incident, two reverted to be safe.
gitlab-com/gl-infra/production#2672 (closed)
What is going on in Sentry Active::RecordNotUniqueError
We have unique constrain in vulnerability_occurrences table, combination of project_id, primary_identifier_id, location_fingerprint, scanner_id has to be unique. I think when we convert location_fingerprint to binary we have some collisions. That violates unique constrain.
How many customers effected
190
Will there be data loss
No, duplicates needs to be cleaned with additional background migration
What about vulnerability feedback
Feedback and Findings match using feedback_fingerprint