[Feature flag] Cleanup `gitlab_duo_chat_requests_to_ai_gateway`
Summary
This issue is to roll out the feature on production,
that is currently behind the gitlab_duo_chat_requests_to_ai_gateway
feature flag.
Owners
- Most appropriate Slack channel to reach out to:
#g_ai_framework
- Best individual to reach out to: @shinya.maeda
Expectations
What are we expecting to happen?
GitLab-Rails requests to AI Gateway for Anthropoic model inference. It's currently requesting to Anthropic directly, which is a blocker against on-premises support.
See Supporting GitLab Duo (chat) for SM and Dedicated (&11251 - closed) for more information.
What can go wrong and how would we detect it?
If it goes wrong, Duo Chat wouldn't be functional for users.
We will conduct an evaluation against Duo Chat to ensure the quality of the response is not degredated.
Aside from that, we can monitor the traffic in the following links:
- AI Gateway: https://log.gprd.gitlab.net/app/r/s/BxKBI
- AI Gateway(Prometheus): https://dashboards.gitlab.net/d/ai-gateway-main/ai-gateway3a-overview?orgId=1
- Duo Chat (GraphQL): https://log.gprd.gitlab.net/app/r/s/i9ZvB
- Request failure in Sidekiq: https://log.gprd.gitlab.net/app/r/s/vVNIv
- Duration https://log.gprd.gitlab.net/app/r/s/6ALVR
- Duration (Prometheus) https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?orgId=1&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gprd&var-stage=main&var-worker=Llm::CompletionWorker
Release the feature
After the feature has been deemed stable, the clean up should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase.
You can either create a follow-up issue for Feature Flag Cleanup or use the checklist below in this same issue.
-
Create a merge request to remove the gitlab_duo_chat_requests_to_ai_gateway
feature flag. Ask for review/approval/merge as usual. The MR should include the following changes:- Remove all references to the feature flag from the codebase.
- Remove the YAML definitions for the feature from the repository.
- Create a changelog entry.
-
Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before the monthly release was tagged, the feature can be officially announced in a release blog post: /chatops run release check <merge-request-url> <milestone>
-
Close the feature issue to indicate the feature will be released in the current milestone. -
Clean up the feature flag from all environments by running these chatops command in #production
channel:/chatops run feature delete gitlab_duo_chat_requests_to_ai_gateway --dev --ops --pre --staging --staging-ref --production
-
Close this rollout issue.
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set gitlab_duo_chat_requests_to_ai_gateway false