## Summary This issue is to roll out the feature introduced in _MR pending_ (tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/605913) on production, that is currently behind the `duo_workflow_finished_event_delivery` feature flag. The flag moves delivery of the **successful** `@GitLabDuo` / Slack messaging reply from the CI-pipeline-gated `Ci::Workloads::WorkloadFinishedEvent` to the new `Ai::DuoWorkflows::WorkflowFinishedEvent`, which is published the moment the workflow transitions to `:finished` — as soon as the agent's answer is checkpointed, ~90s before the CI pipeline finishes. This depends on AI Gateway MR [`ai-assist!6222`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6222), which defers the workflow FINISH transition until after the terminal answer checkpoint is persisted; **it must be deployed to the GitLab-managed fleet before this flag is enabled** (otherwise a checkpoint-durability race can produce a `no_response`). While the flag is off, success delivery stays on `WorkloadFinishedEvent`, so there is no delivery gap. Failure paths always ride `WorkloadFinishedEvent` regardless of the flag. **Feature flag definition:** - **Name:** `duo_workflow_finished_event_delivery` - **Description:** Deliver the @GitLabDuo/Slack messaging reply from Ai::DuoWorkflows::WorkflowFinishedEvent (fired as soon as the agent's answer is checkpointed) instead of waiting for the CI-pipeline-gated Ci::Workloads::WorkloadFinishedEvent, cutting ~90s off the response time. - **Introduced by:** _MR pending_ - **Milestone:** 19.3 - **Group:** group::code review - **Type:** gitlab_com_derisk - **Default enabled:** false ## Owners - Most appropriate Slack channel to reach out to: `#g_ai_coding` - Best individual to reach out to: @kinsingh ## Expectations ### What are we expecting to happen? When enabled, a successful `@GitLabDuo` reply (MR-mention `gitlab_duo_note` adapter, and Slack) is delivered as soon as the workflow finishes — roughly 90s sooner than today, where the reply waits for the whole CI pipeline hosting the agent to wind down. Nothing changes for failed/stopped/dropped workflows: those never publish `WorkflowFinishedEvent` and continue to be handled by `WorkloadFinishedEvent`. This flag is orthogonal to `gitlab_duo_note_callback_high_urgency`, which selects *which* worker handles the callback; this one selects *which event* triggers success delivery. ### What can go wrong and how would we detect it? - **Checkpoint-durability race:** if enabled before `ai-assist!6222` is deployed to the managed fleet, the finish transition can fire before the answer checkpoint is persisted, so `extract_final_message` finds nothing and the user gets `on_flow_failed(:no_response)` instead of a reply. Mitigated by only enabling after the AIGW change is deployed. - **Double delivery:** the pipeline event fires later for the same workflow. Mitigated in code — `handle_workload_finished` skips success delivery when the flag is on, and never posts a failure over an already-`finished` workflow. - **Delivery regression:** if replies stop posting, disable the flag to fall back to the `WorkloadFinishedEvent` path. Monitor: - Latency/error rates for `Ai::Messaging::CallbackDispatchWorker` / `Ai::Messaging::CallbackWorker` and their queues on https://dashboards.gitlab.net - `no_response` outcomes and exceptions in Sentry for the messaging adapters - User feedback in `#g_ai_coding` ## Rollout Steps Note: This is an instance-wide (`:instance`) flag, so it is enabled globally rather than per actor. Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] Enable the feature globally on non-production environments with `/chatops gitlab run feature set duo_workflow_finished_event_delivery true --dev --pre --staging --staging-ref` - [ ] Verify that a `@GitLabDuo` mention reply is delivered on workflow completion (before the CI pipeline finishes). The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/). - [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). - See [`#e2e-run-staging` Slack channel](https://gitlab.enterprise.slack.com/archives/CBS3YKMGD) and look for the following messages: - test kicked off: `Feature flag duo_workflow_finished_event_delivery has been set to true on **gstg**` - test result: `This pipeline was triggered due to toggling of duo_workflow_finished_event_delivery feature flag` ### Before production rollout - [ ] Confirm the AI Gateway change [`ai-assist!6222`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6222) has been deployed to the GitLab-managed AIGW/DWS fleet before enabling the flag (this is the prerequisite that removes the checkpoint-durability race). ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - Ensure that the feature MRs have been deployed to both production and canary with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] Because this is an `:instance` flag it cannot be scoped to a single namespace; validate on `staging-canary` first (above), then proceed to the global production enable below. ### Preparation before global rollout - [ ] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [ ] Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process). Cross link the issue here if it does. - [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias. - [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/development/feature_flags/controls/#communicate-the-change)). ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Enable the feature globally on production with `/chatops gitlab run feature set duo_workflow_finished_event_delivery true`. - Monitor the appropriate graphs on https://dashboards.gitlab.net for at least 15 minutes afterward. - [ ] After the feature has been enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup) or use the checklist below in this same issue. - [ ] Create a merge request to remove the `duo_workflow_finished_event_delivery` feature flag. Ask for review/approval/merge as usual. The MR should include the following changes: - Make `WorkflowFinishedEvent` the unconditional success-delivery path (remove the `if:` flag guards on its subscriptions). - Remove the `deliver_success ... unless workflow_finished_delivery_enabled?` guard and the `workflow.finished?` skip branch from `handle_workload_finished`, leaving `WorkloadFinishedEvent` to handle failures only. - Remove all references to the feature flag from the codebase and delete its YAML definition. - [ ] Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1), the feature can be officially announced in a release blog post: `/chatops gitlab run release check <merge-request-url> <milestone>` - [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running this chatops command in `#production` channel: `/chatops gitlab run feature delete duo_workflow_finished_event_delivery --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops gitlab run feature set duo_workflow_finished_event_delivery false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set duo_workflow_finished_event_delivery false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops gitlab run feature delete duo_workflow_finished_event_delivery --dev --pre --staging --staging-ref --production ```