## Summary This issue is to roll out [the Secrets Manager enrollment toggles](https://gitlab.com/gitlab-org/gitlab/-/issues/598522) on production, currently behind two feature flags: - `secrets_manager_namespace_enrollment` — gates the per-namespace enrollment toggle in group settings (GitLab.com). - `secrets_manager_instance_enrollment` — gates the instance-wide enrollment toggle in admin settings (GitLab Self-Managed). Both flags ship together in !235078 and are tracked here as a single rollout because they share a DRI, team, and target audience. ## Owners - Most appropriate Slack channel to reach out to: `#g_pipeline-security` - Best individual to reach out to: @iamricecake ## Expectations ### What are we expecting to happen? When enabled, group Owners on GitLab.com see an enrollment toggle in group settings that opts the root namespace into Secrets Manager. Instance admins on Self-Managed see an analogous toggle in admin settings that opts the instance into Secrets Manager. Enrollment is the long-term gate that carries from Open Beta to GA — once enrolled, Secrets Manager remains available regardless of the legacy `secrets_manager` / `group_secrets_manager` flags. ### What can go wrong and how would we detect it? - Enrollment record corruption could leave a namespace stuck enrolled or unenrolled. Mitigation: enroll/unenroll go through `NamespaceEnrollmentService` / `InstanceEnrollmentService` with idempotency + audit events. - Authorization bypass — only Owners (group) or admins (instance) can mutate. Audit events provide a paper trail. - Watch: error rate on `NamespaceSecretsManagerEnroll`, `NamespaceSecretsManagerUnenroll`, `InstanceSecretsManagerEnroll`, `InstanceSecretsManagerUnenroll` GraphQL mutations. ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit>` - [ ] Enable both flags globally on non-production environments: - `/chatops gitlab run feature set secrets_manager_namespace_enrollment true --dev --pre --staging --staging-ref` - `/chatops gitlab run feature set secrets_manager_instance_enrollment true --dev --pre --staging --staging-ref` - [ ] Verify that the enrollment toggles render and that enroll/unenroll mutations succeed end-to-end. ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted to `#g_pipeline-security`. - [ ] Enable for GitLab internal first: - `/chatops gitlab run feature set --group=gitlab-org,gitlab-com secrets_manager_namespace_enrollment true` - [ ] Verify the enrollment toggle is visible to internal Owners and that enrollment persists. ### Preparation before global rollout - [ ] Set a milestone for stable rollout. - [ ] Ensure that documentation exists for the feature and history text has been added per [docs](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text). - [ ] Notify [`#support_gitlab-com`](https://gitlab.slack.com/archives/C4XFU81LG) and `#g_pipeline-security`. ### Global rollout on production - [ ] Incrementally enable on production: - `/chatops gitlab run feature set secrets_manager_namespace_enrollment 50 --actors` - Wait ≥ 15 minutes, monitor dashboards. - `/chatops gitlab run feature set secrets_manager_namespace_enrollment 100` - [ ] After namespace flag at 100%, enable the instance flag globally (note: only relevant on Self-Managed deployments): - `/chatops gitlab run feature set secrets_manager_instance_enrollment true` - [ ] Wait ≥ 1 day before scheduling the cleanup. ### Release the feature - [ ] Create an MR to remove both feature flags. The flags are `wip` and are scaffolding for the UI toggle — enrollment itself is permanent (database-backed), so removal is a clean delete with no data migration. - [ ] Close [the feature issue](https://gitlab.com/gitlab-org/gitlab/-/issues/598522). - [ ] After cleanup MR is deployed, delete the flags from all environments: - `/chatops gitlab run feature delete secrets_manager_namespace_enrollment --dev --pre --staging --staging-ref --production` - `/chatops gitlab run feature delete secrets_manager_instance_enrollment --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] Disable in production: ``` /chatops gitlab run feature set secrets_manager_namespace_enrollment false /chatops gitlab run feature set secrets_manager_instance_enrollment false ``` - [ ] Disable on non-production environments: ``` /chatops gitlab run feature set secrets_manager_namespace_enrollment false --dev --pre --staging --staging-ref /chatops gitlab run feature set secrets_manager_instance_enrollment false --dev --pre --staging --staging-ref ``` - [ ] Delete the flags from all environments: ``` /chatops gitlab run feature delete secrets_manager_namespace_enrollment --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete secrets_manager_instance_enrollment --dev --pre --staging --staging-ref --production ``` Note: Disabling the flags hides the UI toggles but does NOT unenroll any namespace or instance already enrolled — existing enrollments remain effective via the policy condition. To revert enrollment, an Owner/admin must call the unenroll mutation, or rows can be deleted directly from `secrets_manager_namespace_enrollments` / `application_settings.secrets_manager_instance_enrolled` for emergency rollback.