<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=627716) </details> <!--IssueSummary end--> ## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/593976) currently behind the `dap_agent_config_candidate` feature flag. - DRI: @john.mcdonnell - Team Slack channel: `#<slack-channel-of-dri-team>` > [!note] > Process and guidance live in the docs - this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability - it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? `Gitlab::DuoAgentPlatform::Config` reads a project's flow config from its default branch only. A feature branch change cannot reach this code path before it merges. A previous direct update to `.gitlab/duo/agent-config.yml` merged, broke Duo flows in production, and was reverted. This flag adds a second file, `.gitlab/duo/agent-config-candidate.yml`, so a config change can land inert and then be switched on per project. - Impacts every DAP flow running in CI on `gitlab-org/gitlab` the project the flag targets. The flag actor is the project. The candidate file exists only in `gitlab-org/gitlab`, so no other project is affected. - No data loss risk. The flag only changes which YAML file the platform reads. - Failure is deterministic, not statistical. A broken container image or a broken `setup_script` fails on the first flow run. - If a project has no candidate file, the flag falls back to `agent-config.yml`. Enabling it on such a project is a no-op. Signals to watch (DRI: add the dashboard link on https://dashboards.gitlab.net): - [ ] The `duo_workflow_workload_completed` analytics event. Watch the completion rate for the project. - [ ] The `Invalid config file ... -> <errors>` error from `Ai::DuoWorkflows::StartWorkflowService`. A spike means the candidate YAML fails schema validation. - [ ] The duration of the flow's setup step. This shows whether the config's `cache:` block works. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#<slack-channel-of-dri-team>`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). > [!note] > The flag actor is the project, not a percentage of requests. `percentage_of_actors` hashes each actor into a stable bucket. For a single project, this setting is binary: it either includes that project or excludes it. Ramping from 5% to 25% to 50% has no observable effect. This rollout uses actor-targeted steps instead of a percentage rollout. **Non-production** ``` /chatops gitlab run feature set dap_agent_config_candidate true --dev --pre --staging --staging-ref ``` **Production** 1. **`gitlab-org/gitlab`.** This step exposes every DAP flow on the monorepo at once ``` /chatops gitlab run feature set --project=gitlab-org/gitlab dap_agent_config_candidate true ``` ## Before global rollout This flag never goes to 100%. A global enable would make every project on GitLab.com look for a candidate file. The fallback makes it a no-op. The end state is the flag enabled on `gitlab-org/gitlab`, then removed. Confirm the relevant gotchas before that step - see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) - see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Cleanup for this flag has extra steps, because the candidate file must become the real config file: 1. Copy the contents of `.gitlab/duo/agent-config-candidate.yml` into `.gitlab/duo/agent-config.yml`. 2. Delete `.gitlab/duo/agent-config-candidate.yml`. 3. Remove the dual-file loading code and the flag definition from `Gitlab::DuoAgentPlatform::Config`. 4. Then run the release check and delete the flag: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/233094 19.4 /chatops gitlab run feature delete dap_agent_config_candidate --dev --pre --staging --staging-ref --production ``` ## Rollback The flag is enabled per project, so the disable command must name the same project. A bare `feature set ... false` clears the global gate. It does not clear a project gate. See [disabling feature flags](https://docs.gitlab.com/development/feature_flags/controls/#disabling-feature-flags). ``` /chatops gitlab run feature set --project=gitlab-org/gitlab dap_agent_config_candidate false # production /chatops gitlab run feature set dap_agent_config_candidate false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete dap_agent_config_candidate --dev --pre --staging --staging-ref --production # remove every gate ``` The next config read uses the primary file. The cache key carries the flag state, so no deploy is needed and no stale candidate config is served.