- We are currently running Openbao on `v2.5.5`. We need to upgrade to `v2.6.2` - Openbao releases - https://github.com/openbao/openbao/releases/tag/v2.6.2 - Using recovery key to generate a new root token has been moved to a new authenticated path - https://github.com/openbao/openbao/pull/3041 ## Why - Break-glass root token generation is already broken on `v2.5.5` - `sys/generate-root/*` is disabled by default from `v2.5.3`, our config doesn't re-enable it - `sys/generate-root-token/*` in `v2.6.0` is the working replacement - needs a token plus the recovery key, no `sudo` - tested end-to-end on `v2.6.2` - Unblocks https://gitlab.com/gitlab-org/gitlab/-/work_items/608230 and https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/work_items/36 ## What changed from v2.5.5 - Went through every `CHANGES:`/`SECURITY:` entry across `v2.5.5` to `v2.6.2` - none of them touch anything we use and confirmed no Customer impact - No Rails code change, no data migration - Storage formats unchanged: barrier ciphertext identical, the PostgreSQL backend DDL byte-identical, the KV v2 protobuf diff empty, namespace records only add one new bool - Two security advisories land in `v2.6.2`, both off our paths - `GHSA-rh46-vc3j-w2w3` needs inline-authentication headers or `sys/workflows`, `GHSA-g892-p242-8g86` is in `pki`. We send no inline-auth headers, run no workflows, mount no `pki` - The upgrade also restores `sys/generate-root-token/*`, our break-glass path. `sys/generate-root/*` has been disabled by default since `v2.5.3`, so break-glass is already broken on `v2.5.5` today. Follow up Unblocks https://gitlab.com/gitlab-org/gitlab/-/work_items/608230 unblocked by this upgrade. - Seven distinct deprecation entries land in the span. Six are no-impact for us: physical/file storage (we're on PostgreSQL), the HSM distribution, 32-bit ARM Windows builds, architecture-specific container tags (CNG builds its own images), and the built-in auth/kerberos, auth/ldap, auth/radius and secret/ldap plugins moving to openbao-plugins (we use jwt auth and kv secrets). The seventh matters: core/seal deprecates the built-in alicloudkms, awskms, azurekeyvault, gcpckms, ocikms and pkcs11 auto-unseal wrappers in favour of plugins, with removal in v2.7.0. .com runs seal "gcpckms". Will tackle in v2.7.0 upgrade issue. ## Validation done - GDK: cross-version data and encryption compatibility both directions, the `:gitlab_secrets_manager` suite at 1612 examples and 0 failures, a real CI job reading secrets, manual UI create/read/update/delete, and a full rollback round trip - Cloud-native GKE: seeded a `v2.5.5` baseline, upgraded in place to `v2.6.2`, read the data back byte-identical, project and group secrets resolved through the chart-installed runner ## FIPS pin - not tidy-up, it's what keeps the legacy golang-fips build booting - Deleting `patches/internal/0001-pin-go-kms-wrapping-modules-for-fips-compat.patch` is not tidy-up - it's the only thing that makes a legacy `golang-fips` build of `bao` work - CNG keeps that patch on one build path only - the legacy `golang-fips` arm of the `case` in `gitlab-openbao/Dockerfile.build.ubi`. Every other variant already strips it, so the deletion is invisible until someone needs that arm again - Without the patch on a `golang-fips` toolchain, `bao` compiles fine and then fails at boot, on both seals we actually run - `static` (the chart default) dies with `failed to create GCM: cipher: NewGCMWithRandomNonce requires aes.Block`, and `gcpckms` auto-unseal (GitLab.com) hangs with `error decrypting data with envelope: failed to initialize GCM mode` - That's why `fcatteau` asked to hold the CNG bump - https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/3090 - Correction on my earlier reasoning: the 6 hunks fail at `v2.6.2` because the patch's two target modules moved version, not because they're gone. `go-kms-wrapping/v2` is now `v2.8.0` and `wrappers/static/v2` is now `v2.2.0`, both still there. `wrappers/aead/v2` and `entropy/v2` did leave the root `go.mod`, but they only ever appeared as context lines in the patch - Re-authoring the pin at `v2.6.2` looks feasible and small - roughly 6 hunks over 5 files, using a `replace` pin rather than `require`, plus a one-token shim (`static.Type` to `wrapping.WrapperTypeStatic`) in `helper/kmsplugin/wrapper.go` - Being straight about the gap - that candidate patch builds, passes tests and round-trips data under stock Go, but it has never been built or booted under an actual `golang-fips` toolchain. Untested on the path that matters - A `golang-fips` fallback now needs two reverts in lockstep - the OpenBao version and the CNG commit that moved the toolchain. The patch was only ever the cheaper half - Still true from the old version of this section: the hazard genuinely doesn't exist under the native module. Checked by round-tripping `go-kms-wrapping/v2@v2.8.0`'s `EnvelopeEncrypt`/`EnvelopeDecrypt` and `crypto/cipher.NewGCMWithRandomNonce` at `GOFIPS140=v1.0.0`, `GODEBUG=fips140=only` included ## Gaps to watch - `GITLAB_OPENBAO_VERSION` matches the `*_VERSION` glob in `.code-patterns`, so `e2e:test-on-cng` fires, but `.prevent-tier-1`/`pipeline::expedite` block it first - runs on `tier-2`/`tier-3`/merge trains only, never first-push MRs, so a missing `cng-secrets-manager` job isn't broken. Not in `.cng-dependency-patterns` on purpose, see https://gitlab.com/gitlab-org/gitlab/-/work_items/588112 - No upgrade test anywhere (https://gitlab.com/gitlab-org/gitlab/-/work_items/604775) - seed-then-upgrade by hand - Merging step 1 puts every dev's GDK on 2.6.2 immediately - validate first - Post unseal is slower but has margin - `liveness_probe.failure_threshold` went to `10` (2026-08-03), prod replay 31-37s over 16 promotions; `v2.6.2` doesn't touch `vault/namespace_store.go`, so cost matches `v2.6.1`. Still measure on staging - Every `-fips` image before 2026-09-04 had a `bao` with no real FIPS crypto until https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/3010 - Runway lands two changes at once - don't pin an anomaly on the bump - CNG master's `fips`/`ubi` pipelines have failed on `postgresql` nightly since 2026-09-05 - it's the Go `1.26.7` bump, not us; `gitlab-openbao` builds green there ## Follow ups - `.com` doesn't use the chart - `secrets-manager-runway` is a `Dockerfile` plus `config.hcl`, and `config.hcl:63`'s `seal "gcpckms" {}` is where `v2.7.0`'s external `kms` plugin change lands. Needs its own issue - Self-managed's `awskms` path is what the chart can't express as a `plugin` stanza - `static` stays non-deprecated in `v2.6.2` and is the chart's default, so only the opt-in `awskms` seal bites. Needs its own issue - https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/merge_requests/2 follows this rebase, but has conflicts and needs a reviewer - only the `GitLabDuo` bot is left, nothing pushed since April