Exclude per-site migration bookkeeping from Geo LR publication

Douglas Barbosa Alexandrerequested to merge
608271-geo-lr-excluded-tables into master

What does this MR do and why?

On a Geo pair using logical replication, both sites run the same migrations and maintenance crons. A few bookkeeping tables record that work with locally allocated ids, and they were still in the publication. When rows from both sides meet on the subscriber, the apply worker hits duplicate keys and crash-loops until someone drops the tables from the publication by hand. This showed up while validating the upgrade procedure, with a duplicate key on postgres_async_foreign_key_validations_pkey after a resume. postgres_async_indexes has the same problem: it also has a unique index on name, so remapping ids would not fix it.

This MR grows EXCLUDED_TABLES in ee/lib/tasks/gitlab/geo/logical_replication.rb from 3 to 10 tables, all per-site database maintenance bookkeeping:

  • Batched background migration state: batched_background_migrations, batched_background_migration_jobs, and batched_background_migration_job_transition_logs (which has a foreign key to the jobs table)
  • Async DDL trackers written by migrations: postgres_async_indexes, postgres_async_foreign_key_validations
  • Reindexing cron bookkeeping: postgres_reindex_actions, postgres_reindex_queued_actions

The spec for this constant is now an explicit contract: the exact list, and every entry must exist in the schema. GitLab Dedicated's seed job derives its expectations from the live publication and fails on drift, so this list should only change on purpose. The operational-contract comment in the same file explains the groups and how existing pairs pick up the change: run publication:set_tables on the primary to drop the now-excluded tables, then subscription:refresh on the secondary. Rows already copied to the subscriber stay, since they describe migrations already applied to the replicated data.

Stacked on !253453 (merged) (itself on !253311 (merged)), so this MR targets that branch until it merges.

Out of scope: suppressing batched background migration Sidekiq jobs on LR secondaries (#603994 (closed)), and loose_foreign_keys_deleted_records, which has application semantics and needs its own analysis. This is behind the default-off flag geo_postgresql_replication_agnostic; streaming replication is unaffected.

How to set up and validate locally

  1. On a Geo primary, run bundle exec rake gitlab:geo:logical_replication:publication:set_tables and confirm the ten excluded tables are dropped from the publication.
  2. On the secondary, run bundle exec rake gitlab:geo:logical_replication:subscription:refresh.
  3. Run the specs: 
    bundle exec rspec ee/spec/lib/tasks/gitlab/geo ee/spec/tasks/gitlab/geo

publication:set_tables also becomes partition-aware. batched_background_migration_job_transition_logs is partitioned, and its leaves live in gitlab_partitions_dynamic under dated names. A table is now excluded through its root parent, and partitions that other tooling published one by one are dropped with their schema, since Rails does not list them and the loop never reached them before.

Validation:

  • 175 specs green across ee/spec/lib/tasks/gitlab/geo and ee/spec/tasks/gitlab/geo, rubocop clean.
  • End to end on the two-site logical replication rig (https://gitlab.com/dbalexandre/geo-caproni-lr): a pair reseeded with the new list had none of the ten tables (or their partitions) in the publication; set_tables on that publication changed nothing; after adding one excluded table and one partition leaf by hand, set_tables removed exactly those two, the leaf with its schema; with replication paused, the same ids were inserted into the excluded tables on both sides and replication resumed with zero apply errors and no rows crossing, where the previous round had crash-looped. The promotion step of the usual run was skipped this time because the rig's secondary VM lost its network interface; this MR does not touch the promotion path, which passed twice this week on the stacked MRs.

References

Resolves #608271 (closed)

Related:

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports