## Summary Investigate whether it's safe to run OpenBao High Availability (HA) mode considering that we leverage self initialization, and that a race condition in self init has been reported. OpenBao HA is currently disabled for .com (through Runway config) but enabled in the OpenBao chart. This was surfaced when considering the upgrade to v2.5.1, which supports horizontal read scalability. ## Further details On GitLab.com we've limited OpenBao to a single instance due to race conditions during self-init. https://gitlab.com/gitlab-org/govern/secrets-management/gitlab-secrets-manager-container/-/merge_requests/76 However, this is not consistent with the OpenBao chart. - The chart default to `2` replicas. https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/merge_requests/57 - HA mode has been validated as part of operational validation https://gitlab.com/gitlab-org/gitlab/-/work_items/589411#note_3129906883 - HA mode is documented in admin docs. https://gitlab.com/gitlab-org/gitlab/-/work_items/568357#note_2945728711 Some self-init issues might be solved by https://github.com/Ki-Reply-GmbH/openbao/pull/57. Related: https://gitlab.com/gitlab-org/gitlab/-/work_items/582828+ The last two releases of OpenBao don't seem to address race conditions in self init. ## Relevant links GitHub issues: - https://github.com/openbao/openbao/issues/2098 (parallel self-init) - https://github.com/openbao/openbao/issues/2097 (postgresql lock removal detection) - https://github.com/openbao/openbao/issues/2096 (postgresql truncation detection) Relevant MRs: - https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/merge_requests/57+ - https://gitlab.com/gitlab-org/govern/secrets-management/gitlab-secrets-manager-container/-/merge_requests/76+ Relevant discussions: - https://gitlab.com/gitlab-org/cloud-native/charts/openbao/-/merge_requests/57#note_3129968643 - https://gitlab.com/gitlab-org/gitlab/-/work_items/589411#note_3129906883 - https://gitlab.com/gitlab-org/govern/secrets-management/gitlab-secrets-manager-container/-/merge_requests/76#note_3129896632 - https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/work_items/719#note_2887337936 - https://gitlab.com/gitlab-org/gitlab/-/work_items/524750#note_2460809795 - https://gitlab.com/gitlab-org/gitlab/-/work_items/568357#note_2945728711 Relevant documentation: - https://docs.gitlab.com/administration/secrets_manager/#high-availability - https://openbao.org/docs/concepts/ha/ Last OpenBao releases: - https://github.com/openbao/openbao/releases/tag/v2.5.0 - https://github.com/openbao/openbao/releases/tag/v2.5.1 /cc @jmallissery