Skip to content

Add NOT NULL constraint to `uuid` column in `security_findings` table

What does this MR do and why?

According to our guidelines for adding NOT NULL constraints on big tables this MR:

  1. Adds a not-valid NOT NULL constraint
  2. Schedules DropInvalidSecurityFindings background migration for the entire security_findings table in batches of 10 000

Related to #284996 (closed)

Database review

db:migrate and db:rollback

➜ bundle exec rails db:migrate                                                            
== 20211110143306 AddNotNullConstraintToSecurityFindingsUuid: migrating =======
-- current_schema()
   -> 0.0002s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE security_findings\nADD CONSTRAINT check_6c2851a8c9\nCHECK ( uuid IS NOT NULL )\nNOT VALID;\n")
   -> 0.0026s
== 20211110143306 AddNotNullConstraintToSecurityFindingsUuid: migrated (0.0265s) 

➜ bundle exec rails db:rollback       
== 20211110143306 AddNotNullConstraintToSecurityFindingsUuid: reverting =======
-- transaction_open?()
   -> 0.0000s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE security_findings\nDROP CONSTRAINT IF EXISTS check_6c2851a8c9\n")
   -> 0.0018s
== 20211110143306 AddNotNullConstraintToSecurityFindingsUuid: reverted (0.0080s) 

Background migration

Number of rows affected

Unknown, according to !63937 (comment 612763457) we already ran a migration on GitLab.com but on-premises customers can have some of these rows so I opted for a batch size of 10 000 and measured the time required to drop a WHOLE batch.

db:migrate and db:rollback

➜ bundle exec rails db:migrate 
== 20211110151350 ScheduleDropInvalidSecurityFindings: migrating ==============
-- Scheduled 0 DropInvalidSecurityFindings jobs with a maximum of 10000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 0 seconds. Expect all jobs to have completed after 2021-11-10 16:12:07 UTC."
== 20211110151350 ScheduleDropInvalidSecurityFindings: migrated (0.0039s) =====

➜ bundle exec rails db:rollback
== 20211110151350 ScheduleDropInvalidSecurityFindings: reverting ==============
== 20211110151350 ScheduleDropInvalidSecurityFindings: reverted (0.0000s) =====

Dropping 10_000 security_findings

Cold cache: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7295/commands/25953

Warm cache: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7295/commands/25954

Total migration duration

Table size: 405 557 791
Batch size: 10 000
Batches needed: 40 556
Time per batch: ~2 minutes
Total running time: 2*40556 ~= 56 days

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michał Zając

Merge request reports