<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=600564) </details> <!--IssueSummary end--> ### Problem to solve GitLab Geo secondaries that use logical replication can break replication when the application accidentally writes to them. To prevent this, on these secondaries the application should: - Connect as a read-only PostgreSQL user (gitlab_readonly) by default. - Route the small set of Sidekiq workers that legitimately need to write to a dedicated geo_privileged queue. - Run the Sidekiq Deployment listening on geo_privileged with the writable gitlab user. ### Proposal For Kubernetes deployments, the idiomatic mechanism is per-pod credentials: each pod gets the specific username and password it needs from a Kubernetes Secret. There is no `DB_ROLE` env var, no helper script, no `geo-db-credentials.rb` file — that's the Omnibus pattern. The chart only needs to: - Render `database.yml` with the readonly user as the default in the main DB block (and every decomposed DB block) when on a Geo LR secondary. - Allow specific Deployments / Jobs (the dedicated `geo_privileged` Sidekiq Deployment, the migrations Job) to override the username and password file at the container level, so they connect as the privileged user instead. - Auto-render the dedicated `geo_privileged` Sidekiq Deployment alongside operator-defined Sidekiq Deployments. - Drain the `geo_privileged` Sidekiq queue before promotion via a Helm hook, so jobs are not stranded when the dedicated Deployment is removed by the role transition. This issue tracks the chart changes for the per-pod credentials model. ### Implementation plan #### Step 1: Add new Helm values under global.geo.logicalReplicationReadonly File: `values.yaml` (the top-level chart values), inside the existing global.geo block around line 644. ```yaml global: geo: # ... existing fields ... logicalReplicationReadonly: enabled: false readonly: username: "gitlab_readonly" # default; override if needed password: secret: "" # name of a Secret containing the readonly password key: "password" ``` The privileged credentials continue to come from the existing `global.psql.password` Secret — that's already what every pod uses today and remains correct for the geo_privileged Sidekiq Deployment, migrations Job, and toolbox. Update `values.schema.json` (if applicable in the local repo layout) to document the new fields. #### Step 2: Add helper templates in `charts/gitlab/templates/_geo.tpl` Add three helpers: ```yaml {{/* Returns "true" if Geo LR readonly is engaged: geo.enabled, geo.role: secondary, and logicalReplicationReadonly.enabled. */}} {{- define "gitlab.geo.logicalReplicationReadonly.enabled" -}} {{- if and (include "gitlab.geo.secondary" $) $.Values.global.geo.logicalReplicationReadonly.enabled -}} true {{- end -}} {{- end -}} {{/* Secret name holding the readonly user's password. */}} {{- define "gitlab.geo.logicalReplicationReadonly.password.secret" -}} {{- $.Values.global.geo.logicalReplicationReadonly.readonly.password.secret | quote -}} {{- end -}} {{/* Key inside the readonly password Secret. */}} {{- define "gitlab.geo.logicalReplicationReadonly.password.key" -}} {{- default "password" $.Values.global.geo.logicalReplicationReadonly.readonly.password.key | quote -}} {{- end -}} ``` These follow the same naming and shape as the existing `gitlab.geo.secondary` and `gitlab.geo.psql.password.{secret,key}` helpers. Add validation in `templates/_checkConfig_geo.tpl` (existing file): if `logicalReplicationReadonly.enabled` is true but `geo.role` is not secondary, or if the readonly Secret reference is empty, fail with a clear message. **NB:** Verify during implementation that default `.Values.global.geo.logicalReplicationReadonly.enabled false` is the right idiom for handling missing values — the existing `gitlab.geo.secondary` helper at `_geo.tpl:7` uses a similar pattern (`default false .enabled`) but the exact shape may need adjustment for nested-map defaulting. #### Step 3: Render `database.yml` with the readonly user in the default position File: `charts/gitlab/templates/_database.yml.tpl.` The template currently renders `username: {{ template "gitlab.psql.username" $context }}` for every database block. Add a branch: when `gitlab.geo.logicalReplicationReadonly.enabled` returns true AND we are rendering the ConfigMap for a default pod (not the dedicated privileged Deployment, not migrations), substitute the readonly username instead. Two design options for "this ConfigMap is for default pods vs. privileged pods": - Option A: Two ConfigMaps per consuming sub-chart. For sub-charts that need both flavours (notably sidekiq), emit two ConfigMaps: `gitlab-sidekiq-readonly` (default) and `gitlab-sidekiq-privileged` (mounted on the dedicated geo_privileged Deployment). The migrations Job always use the privileged ConfigMap. - Option B: Single ConfigMap with ERB env override. Render database.yml once with `username: <%= ENV['DB_USERNAME'] || '<baked-default>' %>`. Default pods set no env override and use the baked default (readonly). The dedicated geo_privileged Deployment and migrations Job set `DB_USERNAME=gitlab` in their pod spec. Option A is more explicit; Option B is smaller in diff and matches the "per-pod credentials" model the issue describes (no DB_ROLE-like indirection, just env reads). Either way, the change in `_database.yml.tpl` is localised: only the `username:` line changes; the password: line continues to use `File.read` so per-pod Secret projection handles the password swap separately (step 4). _Verify:_ rendering tests with `geo.role: secondary` + readonly enabled vs. disabled, asserting the rendered `username:` lines for all decomposed databases (main, ci, embedding, sec — whatever local.psql contains). #### Step 4: Per-pod Secret selection in deployment specs The default psql password Secret reference is `gitlab.psql.password.secret`. When `gitlab.geo.logicalReplicationReadonly.enabled` is true, default Deployments should mount the readonly Secret instead (under the same path the existing `gitlab.psql.password.file` helper points at), and the privileged Deployments + Job continue to mount the existing Secret. Files to touch: - `charts/gitlab/charts/sidekiq/templates/deployment.yaml` — the projected volume that includes the psql password. Currently around the same area as the existing `gitlab.geo.psql.password.secret` mount (line 350–358). Add a branch: if `gitlab.geo.logicalReplicationReadonly.enabled` AND this Deployment is NOT the dedicated geo_privileged one, project the readonly Secret to the postgres/psql-password path. The privileged geo_privileged Deployment keeps projecting the existing Secret. - `charts/gitlab/charts/webservice/templates/` (the deployment) — same logic; web pods are always readonly under this feature. - `charts/gitlab/charts/migrations/templates/job.yaml` and `_jobspec.yaml` — explicitly project the privileged Secret regardless of the readonly opt-in. No change in the common case; the explicit reference makes it future-proof. - `charts/gitlab/charts/toolbox/templates/deployment.yaml` — readonly (current behaviour). - `charts/gitlab/charts/geo-logcursor/templates/` — readonly (it doesn't write to the main DB). If we go with Option B (env-var override) from step 3, also set `DB_USERNAME: gitlab` env on the privileged Deployment + migrations Job. Default pods set nothing and pick up the baked `gitlab_readonly` default. #### Step 5: Auto-render the dedicated geo_privileged Sidekiq Deployment File: `charts/gitlab/charts/sidekiq/templates/deployment.yaml`. The existing template iterates `$.Values.pods[]` to emit one Deployment per entry. When `gitlab.geo.logicalReplicationReadonly.enabled` returns true, prepend (or append) an additional entry with the fixed shape: ```yaml name: geo-privileged queues: 'geo_privileged' concurrency: 1 # default; overridable via values minReplicas: 1 maxReplicas: 1 ``` Dedupe: if the operator's `sidekiq.pods[]` already includes an entry with `queues: geo_privileged` (or whose name is geo-privileged), skip the auto-rendering. The operator's explicit definition wins. Add new values under `sidekiq.geoPrivileged` for tunability (replicas, resources, concurrency), with sensible defaults. Operators rarely need to touch these. The dedicated Deployment uses the privileged Secret per step 4 (and `DB_USERNAME: gitlab` if Option B from step 3). _Verify:_ rendering tests covering (a) feature off → no extra Deployment, (b) feature on, no operator override → extra Deployment rendered with privileged Secret, (c) feature on, operator already defined a geo_privileged pod → only the operator's Deployment renders. #### Step 6: Pre-upgrade Helm hook to drain the geo_privileged queue New file: charts/gitlab/charts/sidekiq/templates/geo-privileged-drain-job.yaml (or similar). A Helm hook Job that runs before any upgrade affects the geo_privileged Deployment: ```yaml apiVersion: batch/v1 kind: Job metadata: name: {{ template "fullname" . }}-geo-privileged-drain annotations: "helm.sh/hook": pre-upgrade,pre-delete "helm.sh/hook-weight": "-5" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: template: spec: restartPolicy: Never containers: - name: drain image: # toolbox image (has gitlab-rake) command: - gitlab-rake - gitlab:sidekiq:migrate_jobs:retry - gitlab:sidekiq:migrate_jobs:schedule - gitlab:sidekiq:migrate_jobs:queued env: # privileged creds, so it can write to Redis routing tables if needed - name: DB_USERNAME value: gitlab # ... psql password from privileged Secret ... ``` The hook is rendered only when `gitlab.geo.logicalReplicationReadonly.enabled` is true. On promotion (operator flips `geo.role: primary` and runs helm upgrade), the feature evaluates false on the NEW values but the hook runs against the OLD values (standard Helm hook semantics) — draining the queue before the dedicated Deployment is removed. Idempotent: running the hook when there are no jobs to migrate is a no-op. _Verify:_ dry-run a Helm upgrade transitioning from `geo.role:` secondary to primary and confirm the hook Job is part of the planned changes; integration test the actual upgrade in a kind cluster. #### Step 7: Documentation Add a section under `doc/charts/charts/gitlab/` (or wherever the existing Geo docs live) covering: - The four new values (`global.geo.logicalReplicationReadonly.{enabled,readonly.username,readonly.password.secret,readonly.password.key}`). - The default per-pod credential assignment table (webservice + default Sidekiq + toolbox = readonly; dedicated geo_privileged Sidekiq + migrations Job = privileged). - The promotion behaviour: flip `geo.role:` secondary → primary, run helm upgrade. The pre-upgrade hook drains; conditional rendering removes the readonly resources. - The security-boundary statement: the readonly user prevents accidental writes from app code; it does not prevent writes from any pod that mounts the privileged Secret. - Operator responsibility: create the `gitlab_readonly` Postgres user and grants out-of-band on the managed Postgres (the chart does not run DDL against the operator's DB). Document the required SQL — matching what GET https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/merge_requests/1827 provides. #### Step 8: End-to-end check In a kind/minikube cluster: - Install the chart with geo.role: secondary, geo.logicalReplicationReadonly.enabled: true, and the two Secrets pre-created. Verify: - Both Secrets exist and are referenced from the expected Deployments. - The geo-privileged Sidekiq Deployment exists and has the privileged Secret mounted. - The default Sidekiq Deployments + webservice have the readonly Secret mounted. - The migrations Job has the privileged Secret mounted. - `kubectl exec` into a default Sidekiq pod and `bundle exec rails runner 'puts ActiveRecord::Base.connection.execute("SELECT current_user").first'` returns `gitlab_readonly`. - Same in the geo-privileged pod returns `gitlab`. - Enqueue a worker tagged `runs_on_geo_lr_secondary!` (e.g., `Database::PartitionManagementWorker` from the Rails issue) → verify it lands in geo_privileged and is processed by the dedicated Deployment. - Enqueue a regular worker that tries to write → verify it lands in a regular Sidekiq queue, fails with a PG permission error. - Flip `geo.role:` primary in values; run helm upgrade. Verify: - The pre-upgrade hook Job runs and exits successfully. - The geo-privileged Sidekiq Deployment is gone. - The readonly Secret reference is removed (default Deployments now mount the privileged Secret again, since we're no longer a secondary). - The chart converges without rollout errors. - Run `helm upgrade` twice in a row — verify idempotency (no diff on the second).