- Root token generation using the recovery key is documented in two places and works in neither. After https://gitlab.com/gitlab-org/gitlab/-/work_items/608229 the mechanism changes shape, so both need rewriting against the new path and actually testing before we call them correct. - Plan: add a Rake task that generates the root token, and point the docs and runbooks at it. MR - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257329 ## Why - `doc/administration/secrets_manager/recovery_key.md` documents `bao operator generate-root`. That returns `405 unsupported operation` on `v2.5.5-gitlab2`, which is what chart `0.18.1` ships. A customer following it today fails at step one - OpenBao `v2.5.3` turned the unauthenticated `sys/generate-root/*` endpoints off by default (CVE-2026-5807). The replacement in `v2.6.0` is `sys/generate-root-token/*`, which needs a token and the recovery key - `doc/administration/secrets_manager/maintenance.md` routes into the same procedure for the auth recovery, so that is blocked too - `.com` has no root token procedure at all. It was never written - Nobody has run the new flow against a real deployment. It is verified on a local `v2.6.1` binary and nowhere else ## Plan - New Rake task `gitlab:secrets_management:openbao:recovery_key:generate_root_token` - Uses the privileged GitLab JWT for the token and asks for the recovery key at run time - So it works the same whether the key was saved with `recovery_key:store` (GitLab database) or `recovery_key:fetch` (external store, like the 1Password vaults on `.com`) - Replaces the four `bao operator generate-root` steps (OTP, init, submit key, decode) with one command, and needs nothing extra on the host - The root token is for OpenBao operations GitLab is not allowed to perform. This matters most once https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36 narrows the `path "*"` policy - It is not a recovery path for broken JWT auth. The new endpoints need a token, and the GitLab JWT login is the only way we get one ## Scope Self managed admin docs, in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257329: - `recovery_key.md` - rewrite the procedure for the Rake task - `troubleshooting.md` - re-check both entry points read correctly once the destinations are rewritten - `maintenance.md` - rewrite the auth recovery steps. For JWT audience drift the root token route cannot work, because the JWT login is what fails. Validate setting `jwt_audience` back to the stored value as the fix, and document that instead - `doc-locale/` is regenerated by the translation bot, not ours to touch `.com` runbooks: - `runbooks/docs/secrets-manager-gke/README.md` - write the procedure. Change request, Rails console access (on-call SREs directly, others through a Teleport `prod-rails-console-rw` access request), recovery key from 1Password, run the Rake task on the console node, revoke the token - Lands once a GitLab release with the Rake task is deployed on `.com` ## Validate before documenting Nothing gets written up as correct until it has been run: - On a cloud native install with `v2.6.2-gitlab1`, follow the docs exactly as written with the Rake task: store the key, generate the root token, confirm `policies: ["root"]` and a privileged write, revoke it - Failure cases: wrong key, an attempt already in progress, a standby pod - Confirm it works through Rails inline auth. The local `v2.6.1` test used a persistent token from `bao token create`, and Rails never holds one - JWT audience drift: confirm the logins fail, then confirm setting `jwt_audience` back fixes them - `.com` recovery keys already exist in 1Password, created with `recovery_key:fetch` - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22392 and https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21589 - Optional: rehearse the runbook on `.com` staging once the Rake task is deployed there ## Dependencies - Blocked by https://gitlab.com/gitlab-org/gitlab/-/work_items/608229. The endpoint does not exist before `v2.6.0`. `.com` is already on `v2.6.2`, self managed charts and the Rails `GITLAB_OPENBAO_VERSION` bump are still pending. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257329 stays draft until then - Not blocked by https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36. The current policy is `path "*"`, which already covers `sys/generate-root-token/*`. Once `#36` narrows the policy it needs the two paths granted explicitly, and they are already in the policy planned there ## Notes for whoever writes it - No key in our deployment is ever split. Single key, no quorum, no threshold to meet, no partial shares. Write `the recovery key`, not `a share` - share language in a procedure means it was copied from upstream Vault docs rather than written for this deployment - `sys/generate-root-token/*` needs `update` only, and no `sudo` - The generated root token does not expire. Every procedure ends with revoking it - `runbooks` has its own conventions - `markdownlint` `MD060` wants table pipes re-padded by hand, the styleguide is enforced in review, and `@gsgl` merges