Skip to content

Set all NULL `lock_version` values to 0, PART 2

Mario de la Ossa requested to merge 28560_cleanup_optimistic_locking_db_pt2 into master

What does this MR do?

This is part 2 of !18418 (merged)

In a background migration, sets the value for lock_version to 0 if it's nil in the following tables:

  • ci_builds
  • ci_pipelines
  • ci_stages

Output of migration

== 20200217210353 CleanupOptimisticLockingNullsPt2: migrating =================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_stages, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0046s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:ci_stages, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0261s
-- execute("RESET ALL")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_builds, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0103s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:ci_builds, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0070s
-- execute("RESET ALL")
   -> 0.0004s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_pipelines, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0061s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- add_index(:ci_pipelines, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0050s
-- execute("RESET ALL")
   -> 0.0005s
== 20200217210353 CleanupOptimisticLockingNullsPt2: migrated (0.2406s) ========

Time calculations

For each of these I performed EXPLAIN SELECT count(*) table_name WHERE lock_version IS NULL

Batches are 10,000 rows each

CI Builds table
Aggregate  (cost=213383199.17..213383199.18 rows=1 width=8)
  ->  Seq Scan on ci_builds  (cost=0.00..213366227.28 rows=6788756 width=0)
        Filter: (lock_version IS NULL)

6788756/10000 = ~679 batches ~1358 minutes = ~23 hours

CI Pipelines
Aggregate  (cost=12590176.44..12590176.45 rows=1 width=8)
  ->  Seq Scan on ci_pipelines  (cost=0.00..12576426.08 rows=5500145 width=0)
        Filter: (lock_version IS NULL)

5500145/10000 = 551 batches ~1,102 minutes = ~19 hours

CI Stages
Aggregate  (cost=10342082.50..10342082.51 rows=1 width=8)
  ->  Seq Scan on ci_stages  (cost=0.00..10308197.88 rows=13553849 width=0)
        Filter: (lock_version IS NULL)

13553849/10000 = 1,356 batches ~2712 minutes = ~45 hours

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Refs #207306 (closed)

Edited by Mario de la Ossa

Merge request reports