Skip to content

Fix batched migrations using legacy job arguments

What does this MR do?

Some of the PK conversions, using the new batched background migrations, were released first for GitLab.com, and later for self-managed. Meanwhile we changed the format we use for batched_background_migrations.job_arguments. This MR updates the records using the legacy format to what we currently use.

Database migrations

When migrations with both legacy and current format exist

gitlabhq_development=# select id, job_class_name, table_name, column_name, job_arguments from batched_background_migrations where table_name in ('events', 'push_event_payloads') order by id asc;
 id |            job_class_name             |     table_name      | column_name |                 job_arguments                  
----+---------------------------------------+---------------------+-------------+------------------------------------------------
 41 | CopyColumnUsingBackgroundMigrationJob | events              | id          | ["id", "id_convert_to_bigint"]
 42 | CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id    | ["event_id", "event_id_convert_to_bigint"]
 51 | CopyColumnUsingBackgroundMigrationJob | events              | id          | [["id"], ["id_convert_to_bigint"]]
 52 | CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id    | [["event_id"], ["event_id_convert_to_bigint"]]
(4 rows)

Up

$ bundle exec rails db:migrate:up VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrating ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrated (0.0090s) 

Down

$ bundle exec rails db:migrate:down VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverting ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverted (0.0000s) 

When migrations with only the current format exist

gitlabhq_development=# select id, job_class_name, table_name, column_name, job_arguments from batched_background_migrations where table_name in ('events', 'push_event_payloads') order by id asc;
 id |            job_class_name             |     table_name      | column_name |                 job_arguments                  
----+---------------------------------------+---------------------+-------------+------------------------------------------------
 51 | CopyColumnUsingBackgroundMigrationJob | events              | id          | [["id"], ["id_convert_to_bigint"]]
 52 | CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id    | [["event_id"], ["event_id_convert_to_bigint"]]
(2 rows)

Up

$ bundle exec rails db:migrate:up VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrating ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrated (0.0082s) 

Down

$ bundle exec rails db:migrate:down VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverting ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverted (0.0000s) 

When migrations with only the legacy format exist

gitlabhq_development=# select id, job_class_name, table_name, column_name, job_arguments from batched_background_migrations where table_name in ('events', 'push_event_payloads') order by id asc;
 id |            job_class_name             |     table_name      | column_name |               job_arguments                
----+---------------------------------------+---------------------+-------------+--------------------------------------------
 41 | CopyColumnUsingBackgroundMigrationJob | events              | id          | ["id", "id_convert_to_bigint"]
 42 | CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id    | ["event_id", "event_id_convert_to_bigint"]
(2 rows)

Up

$ bundle exec rails db:migrate:up VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrating ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: migrated (0.0081s) 

The records after being migrated:

gitlabhq_development=# select id, job_class_name, table_name, column_name, job_arguments from batched_background_migrations where table_name in ('events', 'push_event_payloads') order by id asc;
 id |            job_class_name             |     table_name      | column_name |                 job_arguments                  
----+---------------------------------------+---------------------+-------------+------------------------------------------------
 41 | CopyColumnUsingBackgroundMigrationJob | events              | id          | [["id"], ["id_convert_to_bigint"]]
 42 | CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id    | [["event_id"], ["event_id_convert_to_bigint"]]
(2 rows

Down

$ bundle exec rails db:migrate:down VERSION=20210615234935
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverting ========
== 20210615234935 FixBatchedMigrationsOldFormatJobArguments: reverted (0.0000s) 

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #331815 (closed)

Edited by Krasimir Angelov

Merge request reports