Skip to content

Drop foreign keys prior to dropping individual detached partitions

What does this MR do and why?

This MR modifies the detached partition dropper so that it drops the foreign keys of the detached table individually, and using the lock_retries strategy to reduce the impact of the locking.

This will reduce the locks taken when the partition is actually dropped, since each foreign key drop requires an access exclusive lock on both involved tables.

The incorrect behavior being fixed here was pointed out in #337155 (comment 701918411), and the overarching issue for partition pruning is #332199 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

To validate this locally, we'll need to set up some old partitions that can be dropped.

  1. Change the line partitioned_by :created_at, strategy: :monthly, retain_for: 3.months in the WebHookLog model to retain_for: 6.months so that we generate some extra partitions.
  2. Enable the relevant feature flags: Feature.enable(:partition_pruning), Feature.enable(:drop_detached_partitions)
  3. Close your console, kill spring if it's running, and gdk stop rails.
  4. Run bin/rails db:reset. This will wipe your local database, but it's necessary to convince the partitioning code to create partitions for the past.
  5. Run gdk psql -c '\d+ web_hook_logs' to view the partitions of the table. You should see partitions back to 2021-04-01 (6 months ago).
  6. Change the retain_for line in WebHookLog back to 3.months
  7. In a new console, sync the partitions of the web_hook_logs table: Gitlab::Database::Partitioning::PartitionManager.new(WebHookLog).sync_partitions
  8. Verify that you only see 3 months of past partitions attached (plus the default 5 into the future) when you run gdk psql -c '\d+ web_hook_logs'.

We now have 3 tables that used to be partitions of the web_hook_logs table, and we want to test that they're dropped correctly.

  1. First, update their drop time to be in the past. In a rails console: Postgresql::DetachedPartition.update_all(drop_after: 1.day.ago) (By default the drop time is next week)
  2. Finally, in a rails console, trigger the dropping for these partitions: Gitlab::Database::Partitioning::DetachedPartitionDropper.new.perform. You'll see the 3 tables gitlab_partitions_dynamic.web_hook_logs_202104, _202105, and _202106 dropped, with their foreign keys to the web_hooks table individually dropped before the tables are dropped.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Simon Tomlinson

Merge request reports