<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=602269) </details> <!--IssueSummary end--> ## Summary This issue is to roll out the feature on production, that is currently behind the `ai_use_messaging_adapter_for_mentions` feature flag. When enabled, **foundational-flow `@mention` triggers** are routed through the `GitlabDuoNote` messaging adapter instead of the legacy `FlowTriggers::CreateNoteService` path. The adapter delivers the same UX as `@GitLabDuo`: - an inline "started session" system note posted in the mention thread, - a **framework-delivered** reply when the workflow finishes (posted by `Ai::Messaging::CallbackWorker`), - correctly-branded activity-log session notes (authored by the flow-trigger service account). This replaces today's self-posted `🔄 Processing… → ✅ has started` comment where the agent posts its own reply. **Scope:** routing applies to any foundational flow wired as a `:mention` trigger (Duo Developer / `developer/v1` is the primary current consumer; others such as `fix_pipeline/v1` qualify if mention-triggered). Non-foundational mention triggers and non-mention triggers (`:assign`, `:pipeline`) stay on the legacy `RunService` path, unchanged. `@GitLabDuo` MR mentions use a separate handler and are not affected by this flag. - Feature flag: `ai_use_messaging_adapter_for_mentions` - Actor: **project** (`Feature.enabled?(:ai_use_messaging_adapter_for_mentions, note.project)`) ## Owners - Most appropriate Slack channel to reach out to: `#g_agent_foundations` - Best individual to reach out to: @thomas-schmidt ## Expectations ### What are we expecting to happen? Users who `@mention` a foundational-flow service account on an issue, merge request, or work item see the new adapter UX: an inline started-session note immediately, then a framework-delivered reply in the same thread when the workflow completes, plus the standard activity-log session notes. Behavior for non-foundational and non-mention triggers is unchanged. ### What can go wrong and how would we detect it? - **No reply after the started note is destroyed (top risk).** On completion, `CallbackWorker#extract_final_message` posts the last `agent` message from the workflow checkpoint. If a foundational flow produces no such message, the started note is destroyed and the user sees only a generic "no response" reply (or nothing). Validate end-to-end on a live flow early in rollout. Detect via: user reports, `Ai::Messaging::CallbackWorker` exceptions in Sentry, and Sidekiq metrics for `feature_category=duo_agent_platform`. - **Service-account authoring/permissions.** Started/reply notes are authored as the flow-trigger service account, pre-provisioned as a project member at consumer creation. If membership is missing, `create_note_on` degrades silently (no note). Detect: missing replies + Sentry. - **Shared adapter refactor shipped un-flagged.** The same MR refactored the shared, ungated adapter path (`Base#trigger` → `with_lifecycle_hooks`, the `GitlabDuoNote` constructor). Those changes affect `@GitLabDuo` MR mentions and Slack **regardless of this flag** — a regression there would not be controlled by toggling it. Watch `@GitLabDuo` reply behavior during the same window. Relevant dashboards on https://dashboards.gitlab.net: the `sidekiq` service filtered to `Ai::Messaging::CallbackWorker` / `feature_category=duo_agent_platform`, and the `rails` web error-rate / latency panels. ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] Deploy the feature flag at a percentage (recommended percentage: 50%) with `/chatops gitlab run feature set ai_use_messaging_adapter_for_mentions 50 --actors --dev --pre --staging --staging-ref` - [ ] Monitor that the error rates did not increase (repeat with a different percentage as necessary). - [ ] Enable the feature globally on non-production environments with `/chatops gitlab run feature set ai_use_messaging_adapter_for_mentions true --dev --pre --staging --staging-ref` - [ ] Verify that the feature works as expected. The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/). - [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). - See [`#e2e-run-staging` Slack channel](https://gitlab.enterprise.slack.com/archives/CBS3YKMGD) and look for the following messages: - test kicked off: `Feature flag ai_use_messaging_adapter_for_mentions has been set to true on **gstg**` - test result: `This pipeline was triggered due to toggling of ai_use_messaging_adapter_for_mentions feature flag` If you encounter end-to-end test failures and are unable to diagnose them, you may reach out to the [`#s_developer_experience` Slack channel](https://gitlab.enterprise.slack.com/archives/C07TWBRER7H) for assistance. Note that end-to-end test failures on `staging-ref` [don't block deployments](https://about.gitlab.com/handbook/engineering/infrastructure/environments/staging-ref/#how-to-use-staging-ref). ### Before production rollout - [ ] If the change is significant and you wanted to announce in [#whats-happening-at-gitlab](https://gitlab.enterprise.slack.com/archives/C0259241C), it best to do it before rollout to `gitlab-org/gitlab-com`. ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Ensure that the feature MRs have been deployed to both production and canary with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] This flag uses a **project** actor — enable for specific projects first: - `/chatops gitlab run feature set --project=gitlab-org/gitlab ai_use_messaging_adapter_for_mentions true` - [ ] Verify that the feature works for the specific actors. ### Preparation before global rollout - [ ] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [ ] Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process). Cross link the issue here if it does. - [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias. - [ ] Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) has been updated. - [ ] Ensure that any breaking changes have been announced following the [release post process](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes) to ensure GitLab customers are aware. - [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/development/feature_flags/controls/#communicate-the-change)). ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. - Example: `/chatops gitlab run feature set ai_use_messaging_adapter_for_mentions <rollout-percentage> --actors`. - Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net. - [ ] After the feature has been 100% enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. - [ ] Create a merge request to remove the `ai_use_messaging_adapter_for_mentions` feature flag. Ask for review/approval/merge as usual. The MR should include the following changes: - Remove all references to the feature flag from the codebase. - Remove the YAML definitions for the feature from the repository. - [ ] Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1), the feature can be officially announced in a release blog post: `/chatops gitlab run release check <merge-request-url> <milestone>` - [ ] Close the feature issue to indicate the feature will be released in the current milestone. - [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running these chatops command in `#production` channel: `/chatops gitlab run feature delete ai_use_messaging_adapter_for_mentions --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops gitlab run feature set ai_use_messaging_adapter_for_mentions false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set ai_use_messaging_adapter_for_mentions false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops gitlab run feature delete ai_use_messaging_adapter_for_mentions --dev --pre --staging --staging-ref --production ```