Skip to content

Cleanup legacy background migration module usages from Partition helpers

What does this MR do and why?

This is the next step in closing #365354 (closed), with this all legacy background migration helper usages are removed from Table Partition helpers.

How to set up and validate locally

  1. Create a dummy table (to partition) with created_at and add some entries.
     CREATE TABLE _test_dummy_source_table (
       id SERIAL NOT NULL PRIMARY KEY,
       content VARCHAR(100),
       created_at timestamptz NOT NULL
     );
    
     INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('apr', '2023/04/01');
     INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('may', '2023/05/01');
  2. Add '_test_dummy_source_table' to ALLOWED_TABLES.
    ALLOWED_TABLES = %w[audit_events web_hook_logs _test_dummy_source_table].freeze
  3. Run the below migration to the partition source table. This will create necessary partitioned_table (_test_dummy_source_table_part_a24b0274e4).
     # frozen_string_literal: true
    
     class PartitionTestDummySourceTable < Gitlab::Database::Migration[2.1]
       include Gitlab::Database::PartitioningMigrationHelpers
    
       disable_ddl_transaction!
    
       def up
         partition_table_by_date :_test_dummy_source_table, :created_at
       end
    
       def down
         # Just testing
       end
     end
  4. Feature.disable(:execute_batched_migrations_on_schedule) to avoid executing batched background migrations in local.
  5. Run a migration to enqueue backfilling batched background migration,
     # frozen_string_literal: true
    
     class BackfillPartitionedTestDummySourceTable < Gitlab::Database::Migration[2.1]
       include Gitlab::Database::PartitioningMigrationHelpers
    
       disable_ddl_transaction!
    
       restrict_gitlab_migration gitlab_schema: :gitlab_shared
    
       def up
         enqueue_partitioning_data_migration :_test_dummy_source_table
       end
    
       def down
         # Just testing
       end
     end
  6. Now we should be able to see the BBM record created with status: 1 (active).
    bm = Gitlab::Database::BackgroundMigration::BatchedMigration.find_by(table_name: '_test_dummy_source_table')
    bm.status
  7. Also check the presence of partitioned table (test_dummy_source_table_part_a24b0274e4), without any records.
    gitlabhq_development=# SELECT * from _test_dummy_source_table_part_a24b0274e4;
      id | content | created_at 
      ----+---------+------------
    (0 rows)
  8. Run a migration to finalize the enqueued batched background migration,
     # frozen_string_literal: true
    
     class FinalizeTestDummySourceTableBackill < Gitlab::Database::Migration[2.1]
       include Gitlab::Database::PartitioningMigrationHelpers
    
       disable_ddl_transaction!
    
       restrict_gitlab_migration gitlab_schema: :gitlab_shared
    
       def up
         finalize_backfilling_partitioned_table :_test_dummy_source_table
       end
    
       def down
         # no op
       end
     end
  9. bm.reload => The migration should be finished now (with status 3).
  10. And test_dummy_source_table_part_a24b0274e4 will be having all data copied.
    gitlabhq_development=# SELECT * from _test_dummy_source_table_part_a24b0274e4;
    id | content |       created_at       
    ----+---------+------------------------
     1 | apr     | 2023-04-01 00:00:00+00
     2 | may     | 2023-05-01 00:00:00+00
    (2 rows)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #365354 (closed)

Edited by Prabakaran Murugesan

Merge request reports