Skip to content

Add single db variant of partitioning rake tasks

Patrick Bair requested to merge pb-per-database-partitioning-tasks into master

What does this MR do and why?

Follow up from !82213 (merged) to update partitioning rake tasks with single-database variants when multiple databases are configured.

With multiple databases configured, partition management can now be run against a single database, like:

$ rails gitlab:db:create_dynamic_partitions:main

Also, for other rake tasks that manage partitions (db:migrate, db:schema:load), this change will only run partition management for the matching database. For example, when running

$ rails db:migrate:ci

it will call rails gitlab:db:create_dynamic_partitions:ci instead of rails gitlab:db:create_dynamic_partitions

How to set up and validate locally

  1. Setup your development environment with multiple databases as described here: https://docs.gitlab.com/ee/development/database/multiple_databases.html#development-setup
  2. Verify that the different rake tasks are present:
    $ rails -T gitlab:db:create_dynamic_partitions
    rails gitlab:db:create_dynamic_partitions       # Create missing dynamic database partitions
    rails gitlab:db:create_dynamic_partitions:ci    # Create missing dynamic database partitions on the ci database
    rails gitlab:db:create_dynamic_partitions:main  # Create missing dynamic database partitions on the main database
  3. Tail the development log file
    $ tail -f log/development.log
  4. Run the ci rake task with:
    $ DISABLE_POSTGRES_PARTITION_CREATION_ON_STARTUP=true GITLAB_USE_MODEL_LOAD_BALANCING=true rails gitlab:db:create_dynamic_partitions:ci
  5. Check the log output, to see that partition management only ran for ci. It should be at the end, after the app is loaded, similar to the following. At this time it will only run for loose_foreign_keys and only for db_config_name:ci.
    Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe.
    Creating scope :opened. Overwriting existing method Epic.opened.
    Creating scope :closed. Overwriting existing method Epic.closed.
    Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc.
    Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc.
      Gitlab::Database::PostgresPartition Load (5.1ms)  SELECT "postgres_partitions".* FROM "postgres_partitions" WHERE (parent_identifier = concat(current_schema(), '.', 'loose_foreign_keys_deleted_records')) ORDER BY "postgres_partitions"."name" ASC /*application:web,db_config_name:ci,line:/lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'*/
      ↳ lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'
      Gitlab::Database::PostgresPartition Load (4.1ms)  SELECT "postgres_partitions".* FROM "postgres_partitions" WHERE (parent_identifier = concat(current_schema(), '.', 'loose_foreign_keys_deleted_records')) ORDER BY "postgres_partitions"."name" ASC /*application:web,db_config_name:ci,line:/lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'*/
      ↳ lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'
      LooseForeignKeys::DeletedRecord Load (0.8ms)  SELECT "loose_foreign_keys_deleted_records"."id", "loose_foreign_keys_deleted_records"."created_at" FROM "loose_foreign_keys_deleted_records" WHERE "loose_foreign_keys_deleted_records"."partition" = 1 ORDER BY "loose_foreign_keys_deleted_records"."id" ASC LIMIT 1 /*application:web,db_config_name:ci,line:/app/models/loose_foreign_keys/deleted_record.rb:20:in `block in <class:DeletedRecord>'*/
      ↳ app/models/loose_foreign_keys/deleted_record.rb:20:in `block in <class:DeletedRecord>'
      Gitlab::Database::PostgresPartition Load (4.1ms)  SELECT "postgres_partitions".* FROM "postgres_partitions" WHERE (parent_identifier = concat(current_schema(), '.', 'loose_foreign_keys_deleted_records')) ORDER BY "postgres_partitions"."name" ASC /*application:web,db_config_name:ci,line:/lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'*/
      ↳ lib/gitlab/database/partitioning/sliding_list_strategy.rb:21:in `map'
       (5.0ms)  SELECT columns.column_default AS default_value
    FROM information_schema.columns columns
    WHERE columns.column_name = 'partition' AND columns.table_name = 'loose_foreign_keys_deleted_records'
     /*application:web,db_config_name:ci,line:/lib/gitlab/database/partitioning/sliding_list_strategy.rb:82:in `current_default_value'*/
      ↳ lib/gitlab/database/partitioning/sliding_list_strategy.rb:82:in `current_default_value'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Patrick Bair

Merge request reports