Introduce Claude 3.5 Haiku in Gitlab-Rails
What does this MR do and why?
This merge request introduces support for the CLAUDE 3.5 HAIKU model in our Duo Chat tools, enhancing our AI capabilities with a more efficient and cost-effective option.
Note The following model updates is placed around a feature flag to enable us to evaluate the efficacy of the model in staging. The following should allows us to perform evaluations in staging to measure improvements to Duo Chat tool within our daily-evals
Implementation Details
- Added support for CLAUDE 3.5 HAIKU in the model selection logic
- Implemented a feature flag
claude_3_5_haiku_rollout
for controlled rollout - Updated the
model
method to handle the new model option
References
Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Example Screenshots of LangSmith Trace of a few tools using CLAUDE_3_HAIKU previously
Tool | LangSmith Trace |
---|---|
issue_reader | ![]() |
epic_reader | ![]() |
merge_reader | ![]() |
Before | After |
---|---|
![]() |
How to set up and validate locally
- Enable the Feature flag
claude_3_5_haiku_rollout
Feature.enable(:claude_3_5_haiku_rollout, user)
- Run the
issue_reader
tool in Duo Chat to see new model used
Example log
024-11-08_17:19:18.36304 gitlab-ai-gateway : 2024-11-08 18:19:18 [info ] 127.0.0.1:52495 - "POST /v2/chat/agent HTTP/1.1" 200 client_ip=127.0.0.1 client_port=52495 content_type=application/x-ndjson; charset=utf-8 correlation_id=01JC6CK3PVQJ3HK2V16KE68NME cpu_s=0.027322999999999986 duo_chat.agent_available_tools=['epic_reader', 'issue_reader', 'merge_request_reader', 'ci_editor_assistant', 'gitlab_documentation'] duo_chat.agent_tool_action=issue_reader duration_request=0.006379127502441406 duration_s=1.9949571670149453 enabled_feature_flags=expanded_ai_logging first_chunk_duration_s=1.9941708749975078 gitlab_duo_seat_count=100 gitlab_feature_enabled_by_namespace_ids= gitlab_global_user_id=hxOxGNEDbyD789OGQ3GnDe85HzwdQxebqTqjQJQ5jzs= gitlab_host_name=127.0.0.1 gitlab_instance_id=046c9556-092c-4c75-a42d-231b15cf19f2 gitlab_language_server_version=None gitlab_realm=saas gitlab_saas_duo_pro_namespace_ids=None gitlab_version=17.6.0 http_version=1.1 meta.feature_category=duo_chat method=POST path=/v2/chat/agent request_arrived_at=2024-11-08T17:19:16.367799+00:00 response_start_duration_s=0.0024274999741464853 status_code=200 tracked_internal_events=['request_ask_issue', 'request_duo_chat'] url=http://0.0.0.0:5052/v2/chat/agent user_agent=Ruby
2024-11-08_17:19:18.38984 gitlab-ai-gateway : CRITICAL:codesuggestions:Auth is disabled, all users allowed
2024-11-08_17:19:18.39047 gitlab-ai-gateway : 2024-11-08 18:19:18 [debug ] codegen anthropic call: correlation_id=01JC6CK3PVQJ3HK2V16KE68NME max_tokens=4096 stop_sequences=['\n\nHuman', 'Observation:'] temperature=0.1 timeout=Timeout(connect=5.0, read=30.0, write=30.0, pool=30.0) top_k=NOT_GIVEN top_p=NOT_GIVEN
2024-11-08_17:19:18.39114 gitlab-ai-gateway : 2024-11-08 18:19:18 [info ] Request to LLM correlation_id=01JC6CK3PVQJ3HK2V16KE68NME request_content_json={'max_tokens': 4096, 'messages': [{'role': 'user', 'content': 'Please identify the author of #42 issue'}, {'role': 'assistant', 'content': 'The user is asking about the author of issue #42, and they are currently viewing an issue page. I should use the issue_reader tool to get more information about this specific issue.\n```json\n {\n "ResourceIdentifierType": "'}], 'model': 'claude-3-5-haiku-20241022', 'stop_sequences': ['\n\nHuman', 'Observation:'], 'stream': True, 'system': 'You can fetch information about a resource called: an issue.\nAn issue can be referenced by url or numeric IDs preceded by symbol.\nAn issue can also be referenced by a GitLab reference. A GitLab reference ends with a number preceded by the delimiter # and contains one or more /.\nResourceIdentifierType can only be one of [current, iid, url, reference].\nResourceIdentifier can be number, url. If ResourceIdentifier is not a number or a url, use "current".\nWhen you see a GitLab reference, ResourceIdentifierType should be reference.\n\nMake sure the response is a valid JSON. The answer should be just the JSON without any other commentary!\nReferences in the given question to the current issue can be also for example "this issue" or "that issue",\nreferencing the issue that the user currently sees.\nQuestion: (the user question)\nResponse (follow the exact JSON response):\n```json\n{\n "ResourceIdentifierType": <ResourceIdentifierType>\n "ResourceIdentifier": <ResourceIdentifier>\n}\n```\n\nExamples of issue reference identifier:\n\nQuestion: The user question or request may include https://some.host.name/some/long/path/-/issues/410692\nResponse:\n```json\n{\n "ResourceIdentifierType": "url",\n "ResourceIdentifier": "https://some.host.name/some/long/path/-/issues/410692"\n}\n```\n\nQuestion: the user question or request may include: #12312312\nResponse:\n```json\n{\n "ResourceIdentifierType": "iid",\n "ResourceIdentifier": 12312312\n}\n```\n\nQuestion: the user question or request may include long/groups/path#12312312\nResponse:\n```json\n{\n "ResourceIdentifierType": "reference",\n "ResourceIdentifier": "long/groups/path#12312312"\n}\n```\n\nQuestion: Summarize the current issue\nResponse:\n```json\n{\n "ResourceIdentifierType": "current",\n "ResourceIdentifier": "current"\n}\n```\n\nBegin!\n', 'temperature': 0.1} request_method=POST request_url=URL('https://api.anthropic.com/v1/messages') source=ai_gateway.models.base
Numbered steps to set up and validate the change are strongly suggested.
Merge request reports
Activity
added automation:ml feature flag labels
assigned to @nateweinshenker
added pipelinetier-1 label
- A deleted user
added backend featureaddition groupai framework typefeature labels
- Resolved by Nathan Weinshenker
2 Warnings featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart.
For more information, see:
- The Handbook page on merge request types.
- The definition of done documentation.
This merge request does not refer to an existing milestone. Reviewer roulette
Category Reviewer Maintainer backend @serenafang
(UTC-6, 7 hours behind author)
@wandering_person
(UTC+7, 6 hours ahead of author)
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded devopsai-powered sectiondata-science labels
- Resolved by Nathan Weinshenker
added 1 commit
- d3417323 - Create a feature flag passing through model in option parameters
added 1 commit
- 93e92da7 - Create feature flag for rollout of claude gitlab-rails
requested review from @lesley-r
- Resolved by Michael Becker
requested review from @evakadlecova
removed review request for @evakadlecova
requested review from @evakadlecova and removed review request for @lesley-r
added pipeline:mr-approved label
added pipelinetier-2 label and removed pipelinetier-1 label
Before you set this MR to auto-merge
This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3. We will trigger a new pipeline for each transition to a higher tier.
Before you set this MR to auto-merge, please check the following:
- You are the last maintainer of this merge request
- The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
- This pipeline is recent enough (created in the last 8 hours)
If all the criteria above apply, please set auto-merge for this merge request.
See pipeline tiers and merging a merge request for more details.
- Resolved by Michael Becker
@nateweinshenker The code looks good to me, and I verified it locally.
@wandering_person Can you please do the maintainer review?
requested review from @wandering_person
added pipelinetier-3 label and removed pipelinetier-2 label
E2E Test Result Summary
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 7647a899expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Verify | 45 | 0 | 2 | 0 | 47 | ✅ | | Package | 24 | 0 | 11 | 0 | 35 | ✅ | | Create | 129 | 0 | 22 | 0 | 151 | ✅ | | Data Stores | 33 | 0 | 1 | 0 | 34 | ✅ | | Plan | 76 | 0 | 0 | 0 | 76 | ✅ | | Manage | 1 | 0 | 1 | 0 | 2 | ✅ | | Govern | 75 | 0 | 3 | 0 | 78 | ✅ | | Secure | 4 | 0 | 0 | 0 | 4 | ✅ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Fulfillment | 2 | 0 | 0 | 0 | 2 | ✅ | | Ai-powered | 0 | 0 | 1 | 0 | 1 | ➖ | | Release | 5 | 0 | 0 | 0 | 5 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 404 | 0 | 41 | 0 | 445 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+
e2e-test-on-cng:
test report for 7647a899expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Govern | 84 | 0 | 9 | 3 | 93 | ✅ | | Plan | 86 | 0 | 8 | 0 | 94 | ✅ | | Create | 139 | 0 | 21 | 0 | 160 | ✅ | | Data Stores | 33 | 0 | 10 | 0 | 43 | ✅ | | Monitor | 8 | 0 | 12 | 0 | 20 | ✅ | | Verify | 50 | 0 | 15 | 0 | 65 | ✅ | | Manage | 1 | 0 | 9 | 0 | 10 | ✅ | | Package | 24 | 0 | 14 | 0 | 38 | ✅ | | Fulfillment | 2 | 0 | 7 | 0 | 9 | ✅ | | Secure | 1 | 0 | 6 | 0 | 7 | ✅ | | ModelOps | 0 | 0 | 1 | 0 | 1 | ➖ | | Release | 5 | 0 | 1 | 0 | 6 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Ai-powered | 0 | 0 | 2 | 0 | 2 | ➖ | | Configure | 0 | 0 | 3 | 0 | 3 | ➖ | | Growth | 0 | 0 | 2 | 0 | 2 | ➖ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 435 | 0 | 120 | 3 | 555 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+
- Resolved by Nathan Weinshenker
- Resolved by Michael Becker
added 1 commit
- 7647a899 - Claude 3.5 Haiku model addition gitlab-rails
- Resolved by Michael Becker
changed milestone to %17.6
added this merge request to the merge train at position 2
mentioned in commit 5a82afb6
added workflowstaging-canary label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
mentioned in issue gitlab-org/quality/triage-reports#20952 (closed)
added releasedpublished label and removed releasedcandidate label
mentioned in issue gitlab-org/quality/triage-reports#21565 (closed)
mentioned in issue gitlab-org/quality/triage-reports#22058 (closed)