Loading
Add Rake task to scope secrets_manager app policy
Why this MR ?
- Part of the fix for https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36, done in public per https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36#note_4003135718
- The
secrets_managerOpenBao policy behind the privileged GitLab Rails JWT grants every capability, includingsudo, on every path, so that JWT is equivalent to a root token - Removing
sudoalone does not fix this: withpath "*"the token can rewrite its own policy and givesudoback - OpenBao creates the policy only at first init, so existing installs need a way to apply a new one
- Staging change request
- Production change request
What does this MR do ?
- Adds
ee/lib/secrets_management/secrets_manager_policy.hclwith explicit allow rules for the paths Rails calls, plus deny rules for the root namespace paths that the+/+/+/tenant rules would otherwise also match - Adds the
gitlab:secrets_management:openbao:secrets_manager_policy:updateRake task to apply it on existing installs, and to apply later versions of it. Dry run by default - The task refuses OpenBao below 2.6.0, and refuses when OpenBao has no recovery key. Once the policy is applied, the privileged JWT can't change it, so the task asks for a root token generated from the recovery key
- The task does not revoke the root token, so admins can reuse it for other maintenance. It warns at the end to revoke it
- Runs the Secrets Manager specs against this policy instead of a wildcard test policy
- Docs: the maintenance page, the 19.6 upgrade note and the Rake task list
Notes for reviewers
- Why the policy denies
sys/*,identity/*,secrets/*andauth/*? The policy lives in the root namespace, so OpenBao checks tenant requests against full paths likeorg_1/group_2/project_3/sys/auth/x. The+/+/+/rules are there to match those paths, but they also match root paths such assys/policy/a/sys/policies/acl/b. The deny rules start with a longer literal prefix, so they win on root paths and do not affect tenant paths - All Secrets Manager specs now run against
secrets_manager_policy.hcl. The exception issecrets_manager_client_spec.rb, which uses a test-onlyunrestrictedrole because it tests client methods directly in the root namespace. Itsthe secrets_manager policyblock checks the real policy withsys/capabilities-self - gitlab-org/cloud-native/charts/openbao!85 (merged) has a copy of this policy for new installs. Change both files together. Nothing checks that they match yet, gitlab-org/cloud-native/charts/openbao#49 (closed) tracks that
- The task refuses to run if OpenBao has no recovery key. If OpenBao has one but the GitLab database does not, the task asks the operator to confirm they have it, for installs that keep the key outside GitLab
- Known limits
- The token can still create namespaces and write policies inside tenant namespaces. Those policies cannot grant anything in the root namespace
- Until OpenBao has a recovery key, the token can create one with
sys/rotate/recovery/initand use it to get a root token.recovery_key:storeneeds that path, so it stays. The docs say to store the key right after OpenBao is initialized
All scenarios tested on GDK and self managed CNG
- Secrets Manager unit and request specs, with 0
permission denied - GDK: Secrets Manager QA E2E suite, before the
sys/*andidentity/*change all passed, plus project and group provisioning, secrets, permissions, rotation reminders,backfill_api_authand deprovisioning - GKE: the documented
kubectl execcommands, project and group secret reads from CI on the Kubernetes runner, and a fresh chart install - Rollback: recovery key, then root token, then the old policy written back, on GDK and GKE
- Fresh OpenBao with no recovery key: the task refuses
- Rerun with a changed policy on GDK: the JWT write is denied, the task asks for a root token and writes with it. A wrong or empty token writes nothing
How to set up and validate locally
- Dry run, which prints the current and the new policy and writes nothing:
bundle exec rake "gitlab:secrets_management:openbao:secrets_manager_policy:update" - Apply it, then run it again to see
The secrets_manager policy is already up to date:bundle exec rake "gitlab:secrets_management:openbao:secrets_manager_policy:update" DRY_RUN=false - Add a rule to
secrets_manager_policy.hcland run it again withDRY_RUN=false. OpenBao denies the JWT write, so the task asks for a root token. Generate one withgitlab:secrets_management:openbao:root_token:generate, then revoke it as the task's last line says - Provision a project, create a secret and read it from a CI job. Everything keeps working, and the OpenBao audit log shows no
permission deniedfor thesecrets_managerpolicy
After this MR is merged
- Review and merge gitlab-org/cloud-native/charts/openbao!85 (merged)
- Execute Staging change request - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23290
- Execute Production change request - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23291
- Use the scoped policy in GDK - gitlab-development-kit#3387 (closed)
- Check the chart copy of the policy in CI - gitlab-org/cloud-native/charts/openbao#49 (closed)
References
- Issue - https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36
- Chart MR - gitlab-org/cloud-native/charts/openbao!85 (merged)
- Staging change request - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23290
- Production change request - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23291
- GDK follow-up - gitlab-development-kit#3387 (closed)
- Policy drift check follow-up - gitlab-org/cloud-native/charts/openbao#49 (closed)
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Jayakrishnan Mallissery