## Problem Geo secondaries can use PostgreSQL logical replication (LR) instead of streaming replication, behind the flag `geo_postgresql_replication_agnostic` (Phase 1 epic: https://gitlab.com/groups/gitlab-org/-/work_items/23255). Because LR carries no DDL, the secondary applies GitLab migrations itself. That upgrade order and rake support landed in https://gitlab.com/gitlab-org/gitlab/-/work_items/608269 (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253453), and the per-site migration bookkeeping (batched background migration tables, async DDL trackers, reindexing state) left the publication in https://gitlab.com/gitlab-org/gitlab/-/work_items/608271 (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253487). That per-site bookkeeping creates a new hazard. The subscriber's database is writable, and a post-deployment migration running there can record a batched background migration (BBM) locally. The subscriber's own Sidekiq cron then executes that BBM against tables whose rows are also arriving from the primary through replication, producing duplicate writes and unique-constraint violations. Nothing today gates this: the BBM chain (`Database::BatchedBackgroundMigrationWorker`, `SingleDatabaseWorker`, `ExecutionWorker`) only checks the `execute_batched_migrations_on_schedule` ops flag and the DDL kill switch. The `Geo::SkipSecondary` worker concern is only wired up for Elasticsearch workers. The read-only application role (https://gitlab.com/gitlab-org/gitlab/-/work_items/600560) doesn't exist yet. Under streaming replication this was never a problem, since the standby simply rejected writes. ## Findings - `finalize_batched_background_migration` and `ensure_batched_background_migration_is_finished`, when called from a post-deployment migration, run an unfinished BBM inline and synchronously. Suppressing the Sidekiq-driven workers alone does not close the hole. - A survey in https://gitlab.com/gitlab-org/gitlab/-/work_items/603994 found 252 of 1048 post-deployment migrations contain DML: 218 schedule or finalize BBMs, and 34 do direct data changes (`update_all`, deletes, backfills). All of these currently run on the subscriber during upgrades. ## Proposal Mirror what streaming replication already gives us for free: schema changes apply locally, data changes arrive through replication. 1. **BBMs on an LR secondary.** `queue_batched_background_migration` should record the migration as already finished on the secondary, since its data effect will arrive via replication anyway. `finalize_batched_background_migration` and `ensure_batched_background_migration_is_finished` become no-ops there. The three BBM cron workers and the execution worker should skip when `Gitlab::Geo::LogicalReplication.in_use?`. This keeps `schema_migrations` in parity, which both `metadata:verify` and the upgrade flow depend on. 2. **Direct DML in post-deployment migrations.** There's no mechanical way to skip this today: migration files mix DDL and DML, and nothing marks which parts are data changes. UPDATEs applied both locally and again from the publisher converge to the same result. DELETEs of rows already missing are skipped by the replication apply worker. The real risk is INSERT-type DML and anything hitting a unique index. We need to decide between a declarative marker migration authors add to opt out of running data changes on LR secondaries, a migration-context query analyzer that fails the migration on unmarked DML, or accepting double-applied UPDATE/DELETE and only linting against INSERT-type DML. This needs Database team input. 3. **Not an option:** skipping post-deployment migrations wholesale on the subscriber. They carry DDL the next release depends on, and `metadata:verify` would report the subscriber as permanently behind. ## Acceptance criteria - [ ] BBM records created by post-deployment migrations on an LR secondary never execute there, neither through Sidekiq nor through inline finalization, and `schema_migrations` stays in parity with the primary. - [ ] A decision is recorded for direct DML in post-deployment migrations, with the marker or lint implemented if that option is chosen. - [ ] With the flag off, the streaming replication path is unchanged. - [ ] Validated on a live LR pair by upgrading through a release that queues and finalizes a BBM. ## Related - https://gitlab.com/gitlab-org/gitlab/-/work_items/603994 (original BBM issue; this issue carries its scope inside Phase 1) - https://gitlab.com/gitlab-org/gitlab/-/work_items/600560 (read-only application role) - https://gitlab.com/gitlab-org/gitlab/-/work_items/596967 (migrations on an LR secondary, closed investigation) - https://gitlab.com/gitlab-org/gitlab/-/work_items/596973 (read-only access investigation, closed) - https://gitlab.com/gitlab-org/gitlab/-/work_items/608269 (upgrade order) - https://gitlab.com/gitlab-org/gitlab/-/work_items/608271 (excluded tables)