Skip to content

Use BBM in TableManagementHelpers#enqueue_partitioning_data_migration

What does this MR do and why?

This is the initial iteration of #365354 (closed). It avoids using legacy background migration helpers instead uses batched background migrations.

How to set up and validate locally

We can create a dummy source table (without partitions) and follow partitioning steps to test the backfilling logic (which is changed in this MR).

  1. Create a dummy table with created_at and add some entries.

    CREATE TABLE _test_dummy_source_table (
      id SERIAL NOT NULL PRIMARY KEY,
      content VARCHAR(100),
      created_at timestamptz NOT NULL
    );
    
    INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('apr', '2023/04/01');
    INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('may', '2023/05/01');
  2. Create a migration to partition the above table,

     # frozen_string_literal: true
    
     class PartitionTestDummySourceTable < Gitlab::Database::Migration[2.1]
       include Gitlab::Database::PartitioningMigrationHelpers
    
       disable_ddl_transaction!
    
       def up
         partition_table_by_date :_test_dummy_source_table, :created_at
       end
    
       def down
         # Just testing
       end
     end
    
  3. Create a migration to enqueue backfilling batched background migration

     # frozen_string_literal: true
    
     class BackfillPartitionedTestDummySourceTable < Gitlab::Database::Migration[2.1]
       include Gitlab::Database::PartitioningMigrationHelpers
    
       disable_ddl_transaction!
    
       restrict_gitlab_migration gitlab_schema: :gitlab_shared
    
       def change
         enqueue_partitioning_data_migration :_test_dummy_source_table
       end
     end
  4. Add '_test_dummy_source_table' to ALLOWED_TABLES.

    ALLOWED_TABLES = %w[audit_events web_hook_logs _test_dummy_source_table].freeze
  5. Run above (regular) migrations (2) and (3).

  6. Run the enqueued batched migration,

    > bm = Gitlab::Database::BackgroundMigration::BatchedMigration.find_by(table_name: '_test_dummy_source_table')
    > bmr = Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.new(ApplicationRecord.connection)
    > bmr.run_entire_migration(bm)
    # We can find INSERT commands on _test_dummy_source_table_part_<hash>
  7. Check the entries being copied to the partitioned table,

    # Same logic as TableManagementHelpers#make_partitioned_table_name
    > part_hash = Digest::SHA256.hexdigest("_test_dummy_source_table_part").first(10)
    > ApplicationRecord.connection.exec_query("SELECT * FROM _test_dummy_source_table_part_#{part_hash}").rows
    => [[1, "apr", 2023-04-01 00:00:00 +0000], [2, "may", 2023-05-01 00:00:00 +0000]]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #379126 (closed)

Edited by Prabakaran Murugesan

Merge request reports