## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/622600) currently behind the `dap_workspace_agents` feature flag. - DRI: @achueshev - Team Slack channel: `#dev_agent_developer` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) > [!important] > The flag is currently typed `wip`, which cannot be enabled anywhere. It has to be re-typed to `gitlab_com_derisk` or `beta` before any of the commands below are run. This issue is opened ahead of that so the rollout has a home. ## What could go wrong? Blast radius on the Rails side is small: the flag only adds its own name to the `x-gitlab-enabled-feature-flags` header that Rails sends to the AI Gateway, and it changes nothing when off. The behaviour it unlocks lives in the gateway, which binds customer-authored workspace agents into a foundational flow as subagents. Two things to weigh before enabling broadly: - **Customer-authored prompts and toolsets execute as subagents.** A flow that previously ran only agents authored in the gateway repository will start running definitions the customer controls. - **Remote (CI) execution.** Agent definitions are expected to be read from the default branch only, so an agent added on a merge request branch is not picked up by sessions running on that branch. Confirm that holds before enabling for CI execution — see the open question on https://gitlab.com/gitlab-org/gitlab/-/work_items/622600. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#dev_agent_developer`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set dap_workspace_agents 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set dap_workspace_agents true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set dap_workspace_agents <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss dap_workspace_agents true /chatops gitlab run feature set --group=gitlab-org,gitlab-com dap_workspace_agents true /chatops gitlab run feature set --user=achueshev dap_workspace_agents true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253760 19.4 /chatops gitlab run feature delete dap_workspace_agents --dev --pre --staging --staging-ref --production ``` Remove the flag in the AI Gateway repository **before** removing it here. Deleting the Rails definition first makes the gateway fall back to the flag's default (off) immediately, for every instance. ## Rollback ``` /chatops gitlab run feature set dap_workspace_agents false # production /chatops gitlab run feature set dap_workspace_agents false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete dap_workspace_agents --dev --pre --staging --staging-ref --production # remove entirely ``` ## Related - Feature issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/622600 - Introducing merge request: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253760 - AI Gateway implementation: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6694