Skip to content

Set all NULL `lock_version` values to 0, Part 1

Mario de la Ossa requested to merge 28560_cleanup_optimistic_locking_db into master

What does this MR do?

In a background migration, sets the value for lock_version to 0 if it's nil in the following tables:

  • epics
  • issues
  • merge_requests

There is no real way to roll this back (and neither would we want to, as we'd just be setting a value to nil!). A major concern is how much time this would take to execute on .com, and if it requires downtime or not

Output of Migration

== 20200128210353 CleanupOptimisticLockingNulls: migrating ====================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:epics, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0050s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:epics, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0126s
-- execute("RESET ALL")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:merge_requests, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0099s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:merge_requests, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0057s
-- execute("RESET ALL")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:issues, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0094s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- add_index(:issues, :lock_version, {:where=>"lock_version IS NULL", :algorithm=>:concurrently})
   -> 0.0058s
-- execute("RESET ALL")
   -> 0.0005s
== 20200128210353 CleanupOptimisticLockingNulls: migrated (0.2246s) ===========

Time calculations

For each of these I performed EXPLAIN SELECT count(*) table_name WHERE lock_version IS NULL

Batches are 10,000 rows each

Issues table
Aggregate  (cost=8126004.15..8126004.16 rows=1 width=8)
  ->  Seq Scan on issues  (cost=0.00..8092817.51 rows=13274657 width=0)
        Filter: (lock_version IS NULL)

13274657/10000 = 1,328 batches 1,328 batches * 2 minutes between each = 2,656 minutes = ~45 hours

Merge Requests table
Aggregate  (cost=9619905.45..9619905.46 rows=1 width=8)
  ->  Seq Scan on merge_requests  (cost=0.00..9560548.16 rows=23742915 width=0)
        Filter: (lock_version IS NULL)

23742915/10000 = 2,375 batches 2,375 batches * 2 minutes between each = 4,749 minutes = ~80 hours

Epics table
Aggregate  (cost=5742.34..5742.35 rows=1 width=8)
  ->  Seq Scan on epics  (cost=0.00..5735.46 rows=2750 width=0)
        Filter: (lock_version IS NULL)

2750/10000 = less than 1. Single batch to go through everything, estimated time ~2 minutes

Database checklist

When adding migrations:

  • Updated db/schema.rb
  • Added a down method so the migration can be reverted
  • Added the output of the migration(s) to the MR body
  • Added tests for the migration in spec/migrations if necessary (e.g. when migrating data)
  • Added rollback procedure. Include either a rollback procedure or description how to rollback changes

When adding or modifying queries to improve performance:

  • Included data that shows the performance improvement, preferably in the form of a benchmark
  • Included the output of EXPLAIN (ANALYZE, BUFFERS) of the relevant queries

When adding foreign keys to existing tables:

  • Included a migration to remove orphaned rows in the source table before adding the foreign key
  • Removed any instances of dependent: ... that may no longer be necessary

When adding tables:

  • Ordered columns based on the Ordering Table Columns guidelines
  • Added foreign keys to any columns pointing to data in other tables
  • Added indexes for fields that are used in statements such as WHERE, ORDER BY, GROUP BY, and JOINs

When removing columns, tables, indexes or other structures:

  • Removed these in a post-deployment migration
  • Made sure the application no longer uses (or ignores) these structures

Refs #28560 (closed)

Edited by Mario de la Ossa

Merge request reports