Use BBM in TableManagementHelpers#enqueue_partitioning_data_migration
What does this MR do and why?
This is the initial iteration of #365354 (closed). It avoids using legacy background migration helpers instead uses batched background migrations.
How to set up and validate locally
We can create a dummy source table (without partitions) and follow partitioning steps to test the backfilling logic (which is changed in this MR).
-
Create a dummy table with created_at and add some entries.
CREATE TABLE _test_dummy_source_table ( id SERIAL NOT NULL PRIMARY KEY, content VARCHAR(100), created_at timestamptz NOT NULL ); INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('apr', '2023/04/01'); INSERT INTO _test_dummy_source_table (content, created_at) VALUES ('may', '2023/05/01');
-
Create a migration to partition the above table,
# frozen_string_literal: true class PartitionTestDummySourceTable < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers disable_ddl_transaction! def up partition_table_by_date :_test_dummy_source_table, :created_at end def down # Just testing end end
-
Create a migration to enqueue backfilling batched background migration
# frozen_string_literal: true class BackfillPartitionedTestDummySourceTable < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers disable_ddl_transaction! restrict_gitlab_migration gitlab_schema: :gitlab_shared def change enqueue_partitioning_data_migration :_test_dummy_source_table end end
-
Add '_test_dummy_source_table' to ALLOWED_TABLES.
ALLOWED_TABLES = %w[audit_events web_hook_logs _test_dummy_source_table].freeze
-
Run above (regular) migrations (2) and (3).
-
Run the enqueued batched migration,
> bm = Gitlab::Database::BackgroundMigration::BatchedMigration.find_by(table_name: '_test_dummy_source_table') > bmr = Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.new(ApplicationRecord.connection) > bmr.run_entire_migration(bm) # We can find INSERT commands on _test_dummy_source_table_part_<hash>
-
Check the entries being copied to the partitioned table,
# Same logic as TableManagementHelpers#make_partitioned_table_name > part_hash = Digest::SHA256.hexdigest("_test_dummy_source_table_part").first(10) > ApplicationRecord.connection.exec_query("SELECT * FROM _test_dummy_source_table_part_#{part_hash}").rows => [[1, "apr", 2023-04-01 00:00:00 +0000], [2, "may", 2023-05-01 00:00:00 +0000]]
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #379126 (closed)