Define the upgrade order for Geo logical replication pairs

Douglas Barbosa Alexandrerequested to merge
608269-geo-lr-upgrade-ordering into master

What does this MR do and why?

Logical replication carries no DDL, so a Geo secondary running on it has to apply GitLab migrations itself. Until now there was no defined order for upgrading such a pair, no answer for what happens to replication while the two schemas differ, and no recovery path if only one side migrates. When the publisher gains a column the subscriber doesn't have yet, the apply worker fails with "missing replicated column" and keeps retrying until the subscriber catches up. Nothing is lost, since the slot keeps the WAL, but it's noisy and easy to misread as a real problem.

This MR adds:

  • Two new rake tasks, subscription:disable and subscription:enable, to pause and resume replication around an upgrade. Both are idempotent and print the next step to run.
  • An updated metadata:verify that knows whether a subscription exists. With one, it tells you whether the publisher or subscriber is ahead and what to run next, and it no longer suggests re-seeding.
  • An "Upgrading an LR pair" section in the operational-contract comment at the top of ee/lib/tasks/gitlab/geo/logical_replication.rb documenting the upgrade order, what happens if you skip the pause, and how to recover.
  • A db:migrate prerequisite that runs the existing sequence sync first when the site is a logical replication secondary. The initial copy leaves subscriber sequences at their start values, so migrations that insert rows failed on primary-key collisions until the sync ran. The gate is Gitlab::Geo::LogicalReplication.in_use?; other sites see no change, and GEO_SKIP_SEQUENCE_SYNC=1 skips it.

The recommended order is:

  1. subscription:disable
  2. Migrate the publisher (regular and post-deployment migrations)
  3. Migrate the subscriber the same way
  4. publication:set_tables on the publisher
  5. metadata:verify to confirm both sides match
  6. subscription:enable
  7. subscription:refresh (must come after enable, since PostgreSQL refuses to refresh a disabled subscription)

Publisher-first matches the Dedicated blueprint, where the customer's publisher upgrades first, and lines up with Geo's existing pause/upgrade/resume flow for streaming replication.

Stacked on !253311 (merged) (it reuses the metadata:verify helpers), so this MR targets that branch until it merges. The feature is behind the default-off geo_postgresql_replication_agnostic flag and these are operator-run tasks; streaming replication is unchanged.

Out of scope: wiring this into Omnibus (gitlab-ctl reconfigure) or Helm (a migrations job on LR secondaries) is tracked in follow-up issues. Partition churn and batched background migrations on LR secondaries are also separate.

Follow-up issues: omnibus-gitlab#10085 (Omnibus) and gitlab-org/charts/gitlab#6648 (Helm chart).

How to set up and validate locally

  1. Export GEO_PUBLISHER_CONNECTION_STRING and GEO_SUBSCRIBER_CONNECTION_STRING.
  2. bundle exec rake gitlab:geo:logical_replication:subscription:disable
  3. Migrate the publisher: bundle exec rake db:migrate (and post-deployment migrations)
  4. Migrate the subscriber the same way
  5. bundle exec rake gitlab:geo:logical_replication:publication:set_tables
  6. bundle exec rake gitlab:geo:logical_replication:metadata:verify
  7. bundle exec rake gitlab:geo:logical_replication:subscription:enable
  8. bundle exec rake gitlab:geo:logical_replication:subscription:refresh

Run the specs with:

bundle exec rspec ee/spec/lib/tasks/gitlab/geo ee/spec/tasks/gitlab/geo

Validation:

  • 162 specs green across ee/spec/lib/tasks/gitlab/geo and ee/spec/tasks/gitlab/geo, rubocop clean.
  • End to end on a two-site logical replication rig (https://gitlab.com/dbalexandre/geo-caproni-lr) against current master. A pair was built at the 19.3.0 schema and upgraded to master in two steps. Step one (49 migrations) ran publisher-first without the pause: the apply worker hit "missing replicated column", metadata:verify reported the publisher ahead, and replication self-healed within a minute of migrating the subscriber. Step two (95 migrations) followed the documented pause procedure with zero apply errors during the window. The subscriber's db:migrate fired the new prerequisite, advanced 25 sequences, was idempotent on a second run, and honoured the skip flag; the primary's did not fire it. Followed by a full promotion with a clean first write.
  • The same run showed that per-site migration bookkeeping tables (postgres_async_indexes, postgres_async_foreign_key_validations, batched_background_migrations, batched_background_migration_jobs) must leave the publication, otherwise both sides' migration inserts collide when replication resumes. That is #608271 (closed) and comes as the next MR on top of this one.

References

Resolves #608269 (closed)

Related:

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Douglas Barbosa Alexandre

Merge request reports