<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Work on this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=600560) - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=600560) </details> <!--IssueSummary end--> ### Problem to solve GitLab Geo secondaries that use logical replication can break replication when the application accidentally writes to them. To prevent this, on these secondaries the application should: - Connect as a read-only PostgreSQL user (gitlab_readonly) by default. - Route the small set of Sidekiq workers that legitimately need to write (DDL, partition management) to a dedicated geo_privileged queue. - Run the Sidekiq process listening on geo_privileged with the writable gitlab user. ### Proposal - A class method to detect "this node is an LR-mode Geo secondary," used to gate the rest of the behaviour so primaries, non-Geo instances, and physical-replication secondaries are not affected. - A built-in Sidekiq routing rule that sends workers tagged for writes-on-secondary to the `geo_privileged` queue regardless of how an operator has configured `sidekiq['routing_rules']`. This mirrors how the mailers queue is currently handled. - Per-child environment injection in `sidekiq-cluster` so the child listening on `geo_privileged` has `DB_ROLE=privileged` set before Rails boots inside that process. The credentials helper that runs at Rails boot reads this env once and picks the right credentials. - Wildcard handling in `sidekiq-cluster` so a `*` queue group does not include `geo_privileged` and therefore doesn't poach jobs from the privileged child. - Startup validation that rejects queue groups that mix privileged and non-privileged queues (a misconfiguration that would silently break readonly enforcement for the non-privileged queues in the group). ### Implementation plan #### Step 1: Add a `PRIVILEGED_QUEUES` constant _File: sidekiq_cluster/sidekiq_cluster.rb_ Define at the top of the Gitlab::SidekiqCluster module: ```ruby PRIVILEGED_QUEUES = ['geo_privileged'].freeze ``` This single source of truth is used by steps 3, 4, and 5. #### Step 2: Inject DB_ROLE=privileged in start_sidekiq _File: sidekiq_cluster/sidekiq_cluster.rb_ In `start_sidekiq`, before the existing `Process.spawn` call, build the env hash and inject `DB_ROLE` when the child's queues intersect `PRIVILEGED_QUEUES`: ```ruby spawn_env = { 'ENABLE_SIDEKIQ_CLUSTER' => '1', 'SIDEKIQ_WORKER_ID' => worker_id.to_s } spawn_env['DB_ROLE'] = 'privileged' if (queues & PRIVILEGED_QUEUES).any? ``` Pass `spawn_env` to `Process.spawn`. The injection must happen at the spawn boundary because the credentials helper runs at Rails boot inside the child and reads `ENV` once — anything set after fork is too late. _Verify:_ spec in `spec/sidekiq_cluster/sidekiq_cluster_spec.rb`. Two cases: queue list contains `geo_privileged` (env includes `DB_ROLE`); queue list does not (env does not). #### Step 3: Exclude privileged queues from `*` wildcard expansion _File: sidekiq_cluster/cli.rb_ Around line 81 (the wildcard expansion in the queue_groups parsing loop), change: ```ruby next available_queues if queues == WILDCARD_MATCH || routing_rules.empty? ``` to: ```ruby if queues == WILDCARD_MATCH || routing_rules.empty? next available_queues - SidekiqCluster::PRIVILEGED_QUEUES end ``` A `*` queue group expands to all queues except privileged ones, so the wildcard child does not compete with the dedicated `geo_privileged` child. _Verify:_ spec in `spec/commands/sidekiq_cluster/cli_spec.rb`. With `routing_rules` containing `geo_privileged` as a destination, a `*` queue group should not include `geo_privileged` in the resulting child's queues. #### Step 4: Validate against mixed-queue groups _File: sidekiq_cluster/cli.rb_ After `queue-group` parsing (and before `start_and_supervise_workers`), iterate over the parsed groups and raise on any that mix privileged and non-privileged queues: ```ruby queue_groups.each do |group| privileged = group & SidekiqCluster::PRIVILEGED_QUEUES non_privileged = group - SidekiqCluster::PRIVILEGED_QUEUES next unless privileged.any? && non_privileged.any? raise CommandError, "Queue group '#{group.join(',')}' mixes privileged (#{privileged}) with " \ "non-privileged (#{non_privileged}) queues. Privileged queues must be in their own queue group." end ``` The env injection from step 3 is per-child, so mixing would silently apply `DB_ROLE=privileged` to non-privileged queues and break readonly enforcement. _Verify:_ spec `in spec/commands/sidekiq_cluster/cli_spec.rb`. Three cases: pure privileged group passes; pure non-privileged passes; mixed group raises `CommandError` with a clear message. #### Step 5: Add the built-in routing rule _File: lib/gitlab/sidekiq_config/worker_router.rb_ In `parse_routing_rules`, conditionally prepend a built-in `RuleEvaluator` before the operator's rules: ```ruby def parse_routing_rules(routing_rules) # ... existing validation ... built_in = if Gitlab::Geo.logical_replication_enabled? [RuleEvaluator.new(WorkerMatcher.new('tags=geo_lr_writable'), 'geo_privileged', nil)] else [] end built_in + routing_rules.map do |rule_tuple| # ... existing parsing ... end end ``` Because route returns on first match, tagged workers always reach `geo_privileged` on LR secondaries regardless of what operator routing rules say. On any other node type the built-in is empty and behaviour is unchanged. _Verify:_ spec in `spec/lib/gitlab/sidekiq_config/worker_router_spec.rb`. With the predicate stubbed true, a worker carrying the `:geo_lr_writable` tag routes to `geo_privileged` even when operator rules try to send it elsewhere. With the predicate stubbed false, the same worker follows the operator's rules normally. #### Step 6: End-to-end integration check With the `:geo_postgresql_replication_agnostic` feature flag enabled and `Gitlab::Geo.secondary?` stubbed true: - Enqueue a worker tagged with `runs_on_geo_lr_secondary!`. Verify it lands in the geo_privileged queue. - Run `sidekiq-cluster` with queue groups `['*', 'geo_privileged']`. Verify two children are spawned: one with `DB_ROLE` unset (the wildcard child, with geo_privileged excluded from its queue list); one with `DB_ROLE=privileged` (the dedicated child). With the feature flag disabled, the same worker follows the operator's routing rules and no env injection happens. ### Investigation work These are required for the feature to be fully usable but don't gate the Omnibus MR. - **Worker audit:** Identify every Sidekiq worker that writes on a Geo secondary and tag with `runs_on_geo_lr_secondary!`. We already know `Database::PartitionManagementWorker` and `Database::DropDetachedPartitionsWorker`; and `Ci::PartitioningWorker` is a candidate not yet tagged. A full audit is needed. - **Cron-allow-list audit** in `ee/lib/gitlab/geo/cron_manager.rb`. Cross-reference the worker audit against the secondary cron allow-list. Tagged workers that run via cron need to join the allow-list when `Gitlab::Geo.logical_replication_enabled?` is true. Blocked on primaries and physical secondaries. - `Sidekiq::Client.push(queue:)` audit. Grep for explicit-queue enqueues that bypass `Gitlab::SidekiqConfig::WorkerRouter`. Document or remove any on write-on-secondary code paths. - Dev docs. Add a section to `doc/development/sidekiq/worker_attributes.md` for `runs_on_geo_lr_secondary!` alongside the existing `Geo::SkipSecondary` section, so future contributors understand the two complementary patterns.