<!-- Title suggestion: [FF] `secrets_manager_defer_group_source_check` -- <short description> --> ## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/633509) currently behind the `secrets_manager_defer_group_source_check` feature flag. - DRI: @iamricecake - Team Slack channel: `#sec-secrets-manager-application` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag changes what happens when a Secrets Manager secret uses a `group/...` source that is not the project's group or one of its parent groups. This happens most often in merge requests from forks. - **Flag off (today):** pipeline creation fails with a YAML error, so no jobs run. - **Flag on:** the pipeline is created. Only the job that needs the secret is dropped when its runner payload is built, with the new `secrets_manager_source_not_accessible` failure reason. The flag actor is the project that runs the pipeline. For a fork merge request, that is the fork, not the upstream project. Blast radius is small. Only pipelines that use a group source outside the project hierarchy behave differently. No secret is exposed, because the job is dropped before it reaches a runner. Specific risks: - **Job pickup errors:** the runner payload now raises an error for these jobs, which job pickup maps to the new failure reason. No new queries. If the mapping ever broke, the job would be dropped with `scheduler_failure` instead and the error would show in Sentry, so watch the Exceptions link below. - **Turning the flag off mid-rollout:** jobs created while the flag was on are no longer dropped with the new failure reason. They reach the runner and fail with the older, less clear `inline auth ... is required` error. This affects only that job, and no secret is exposed. ## Events <!-- Replace feature flag name in the link--> - [Exceptions with secrets_manager_defer_group_source_check:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22secrets_manager_defer_group_source_check:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with secrets_manager_defer_group_source_check:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22secrets_manager_defer_group_source_check:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#sec-secrets-manager-application`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set secrets_manager_defer_group_source_check 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set secrets_manager_defer_group_source_check true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set secrets_manager_defer_group_source_check <percentage> --actors ``` Or target specific actors instead. The actor is a project, so target the fork projects that hit the bug: ``` /chatops gitlab run feature set --project=gitlab-community/gitlab-org/gitlab-runner secrets_manager_defer_group_source_check true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete secrets_manager_defer_group_source_check --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set secrets_manager_defer_group_source_check false # production /chatops gitlab run feature set secrets_manager_defer_group_source_check false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete secrets_manager_defer_group_source_check --dev --pre --staging --staging-ref --production # remove entirely ```