## Summary Roll out [the feature](https://gitlab.com/groups/gitlab-org/-/work_items/21067) currently behind the `dap_software_factory` feature flag. - Introduced by: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263602 - Related work item: https://gitlab.com/gitlab-org/gitlab/-/work_items/634551 - Related work item for the on-finish settings of flow triggers: https://gitlab.com/gitlab-org/gitlab/-/work_items/633112. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/265220 moves them from `ai_flow_trigger_status_on_finish` to this flag. - DRI: @dmishunov - Team Slack channel: `#<slack-channel-of-dri-team>` - Actor: the top-level namespace or the project. The feature is on for a project when the flag is on for the project or for its top-level namespace. See `Gitlab::Ai::SoftwareFactory.enabled?` in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264877. The group pages use the top-level group. > [!note] > Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag controls these parts: - The publication of `MergeRequests::RequestedChangesCloudEvent` after `MergeRequests::UpdateReviewerStateService` sets the `requested_changes` state and removes the approval. - `Ai::Catalog::Flows::ExecuteMergeRequestRequestedChangesWorkflowTriggersWorker`, which subscribes to the event. It starts the flows of the flow triggers on the `merge_request` event, with the `requested_changes` action. - The **Changes requested** action and the **Software Factory** work item filter in the flow trigger form. - The validation of a new `software_factory_enabled` rule in the filter of a flow trigger. - `Ai::FlowTriggers::ApplyOnFinishSettingsWorker`, which subscribes to `Ai::DuoWorkflows::WorkflowFinishedCloudEvent`. When a flow trigger starts a session from a work item event, and the session finishes, the worker applies the on-finish settings of the trigger. It marks the draft merge requests of the session as ready, then sets the work item status. It writes as the composite identity of the session. - The **When the session finishes** section in the flow trigger form. When the flag is on for a project, these risks apply to that project: - A flow trigger on the `merge_request` event without an `action` filter starts its flow on each "requested changes" verdict. Before, it started only on the `created`, `approved`, and `merged` actions. So the number of flow sessions and CI pipelines can increase. - A verdict of GitLab Duo Code Review starts flows only when `ai_flow_trigger_chaining` is also on. The worker starts each flow trigger one time for each review session. - The on-finish settings apply only to flow triggers that have them, and only when `ai_flow_trigger_chaining` is also on. The new work item status can start the flow triggers of the next stage. - A merge request that the worker marks as ready gets a GitLab Duo Code Review, if automatic reviews are on. Watch for these signals while the flag is on: - Exceptions raised from `MergeRequests::UpdateReviewerStateService`. - Errors of `Ai::Catalog::Flows::ExecuteMergeRequestRequestedChangesWorkflowTriggersWorker` in Sidekiq. - A high count of `LeaseTakenError` retries for the worker. - Errors of `Ai::FlowTriggers::ApplyOnFinishSettingsWorker` in Sidekiq. - A high count of log lines with the message `On-finish status not set` or `On-finish merge request not marked as ready`. ## Events - [Exceptions with dap_software_factory:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22dap_software_factory:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with dap_software_factory:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22dap_software_factory:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#<slack-channel-of-dri-team>`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set dap_software_factory 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set dap_software_factory true --dev --pre --staging --staging-ref ``` **Production**: percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set dap_software_factory <percentage> --actors ``` Or target specific actors instead. The feature is on for a project when the flag is on for the project or for its top-level group. Start with a low-impact project: ``` /chatops gitlab run feature set --project=gitlab-org/ai/swf-base-setup dap_software_factory true /chatops gitlab run feature set --group=gitlab-org dap_software_factory true ``` The second command turns on the feature for all projects of `gitlab-org`, the monolith too. A percentage rollout with `--actors` selects projects and groups separately. A project gets the feature when one of its two actors is selected. So the share of projects with the feature is higher than the percentage. Do not use an opt-out. When the flag is on globally, an opt-out of a top-level group does not turn off the feature for a project of that group. Two related flags still use the top-level group as the actor: - A GitLab Duo Code Review verdict starts flows only when `ai_flow_trigger_chaining` is on for the top-level group. The on-finish settings also need it. - The Software Factory marker on work items needs `workplan` for the top-level group. ## Before global rollout Confirm the relevant gotchas before going to 100%. See [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [ ] [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [ ] In [`doc/user/duo_agent_platform/triggers/_index.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/duo_agent_platform/triggers/_index.md), add the `requested_changes` action to the **Merge request** event type, with its version history. Source: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263642 - [ ] In the same page, in "Actions that don't initiate a trigger", add the exception: a GitLab Duo Code Review verdict can run a flow trigger when `ai_flow_trigger_chaining` is enabled. The flow runs for the human who started the review. Source: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263642 - [ ] [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [ ] [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [ ] [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle). See [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> 19.6 /chatops gitlab run feature delete dap_software_factory --dev --pre --staging --staging-ref --production ``` ## Rollback A rollback also turns off the on-finish settings of flow triggers. ``` /chatops gitlab run feature set dap_software_factory false # production /chatops gitlab run feature set dap_software_factory false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete dap_software_factory --dev --pre --staging --staging-ref --production # remove entirely ```