Skip to content
Snippets Groups Projects

Introduce Claude 3.5 Haiku in Gitlab-Rails

All threads resolved!

What does this MR do and why?

This merge request introduces support for the CLAUDE 3.5 HAIKU model in our Duo Chat tools, enhancing our AI capabilities with a more efficient and cost-effective option.

Note The following model updates is placed around a feature flag to enable us to evaluate the efficacy of the model in staging. The following should allows us to perform evaluations in staging to measure improvements to Duo Chat tool within our daily-evals

Implementation Details

  • Added support for CLAUDE 3.5 HAIKU in the model selection logic
  • Implemented a feature flag claude_3_5_haiku_rollout for controlled rollout
  • Updated the model method to handle the new model option

References

Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

Related to #502925 #500421

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Example Screenshots of LangSmith Trace of a few tools using CLAUDE_3_HAIKU previously

Tool LangSmith Trace
issue_reader Issue Reader Trace
epic_reader Epic Reader Trace
merge_reader Merge Reader Trace
Before After
Screenshot_2024-11-08_at_6.22.14_PM

How to set up and validate locally

  1. Enable the Feature flag claude_3_5_haiku_rollout
Feature.enable(:claude_3_5_haiku_rollout, user)
  1. Run the issue_reader tool in Duo Chat to see new model used

Example log

024-11-08_17:19:18.36304 gitlab-ai-gateway     : 2024-11-08 18:19:18 [info     ] 127.0.0.1:52495 - "POST /v2/chat/agent HTTP/1.1" 200 client_ip=127.0.0.1 client_port=52495 content_type=application/x-ndjson; charset=utf-8 correlation_id=01JC6CK3PVQJ3HK2V16KE68NME cpu_s=0.027322999999999986 duo_chat.agent_available_tools=['epic_reader', 'issue_reader', 'merge_request_reader', 'ci_editor_assistant', 'gitlab_documentation'] duo_chat.agent_tool_action=issue_reader duration_request=0.006379127502441406 duration_s=1.9949571670149453 enabled_feature_flags=expanded_ai_logging first_chunk_duration_s=1.9941708749975078 gitlab_duo_seat_count=100 gitlab_feature_enabled_by_namespace_ids= gitlab_global_user_id=hxOxGNEDbyD789OGQ3GnDe85HzwdQxebqTqjQJQ5jzs= gitlab_host_name=127.0.0.1 gitlab_instance_id=046c9556-092c-4c75-a42d-231b15cf19f2 gitlab_language_server_version=None gitlab_realm=saas gitlab_saas_duo_pro_namespace_ids=None gitlab_version=17.6.0 http_version=1.1 meta.feature_category=duo_chat method=POST path=/v2/chat/agent request_arrived_at=2024-11-08T17:19:16.367799+00:00 response_start_duration_s=0.0024274999741464853 status_code=200 tracked_internal_events=['request_ask_issue', 'request_duo_chat'] url=http://0.0.0.0:5052/v2/chat/agent user_agent=Ruby
2024-11-08_17:19:18.38984 gitlab-ai-gateway     : CRITICAL:codesuggestions:Auth is disabled, all users allowed
2024-11-08_17:19:18.39047 gitlab-ai-gateway     : 2024-11-08 18:19:18 [debug    ] codegen anthropic call:        correlation_id=01JC6CK3PVQJ3HK2V16KE68NME max_tokens=4096 stop_sequences=['\n\nHuman', 'Observation:'] temperature=0.1 timeout=Timeout(connect=5.0, read=30.0, write=30.0, pool=30.0) top_k=NOT_GIVEN top_p=NOT_GIVEN
2024-11-08_17:19:18.39114 gitlab-ai-gateway     : 2024-11-08 18:19:18 [info     ] Request to LLM                 correlation_id=01JC6CK3PVQJ3HK2V16KE68NME request_content_json={'max_tokens': 4096, 'messages': [{'role': 'user', 'content': 'Please identify the author of #42 issue'}, {'role': 'assistant', 'content': 'The user is asking about the author of issue #42, and they are currently viewing an issue page. I should use the issue_reader tool to get more information about this specific issue.\n```json\n                    {\n                      "ResourceIdentifierType": "'}], 'model': 'claude-3-5-haiku-20241022', 'stop_sequences': ['\n\nHuman', 'Observation:'], 'stream': True, 'system': 'You can fetch information about a resource called: an issue.\nAn issue can be referenced by url or numeric IDs preceded by symbol.\nAn issue can also be referenced by a GitLab reference. A GitLab reference ends with a number preceded by the delimiter # and contains one or more /.\nResourceIdentifierType can only be one of [current, iid, url, reference].\nResourceIdentifier can be number, url. If ResourceIdentifier is not a number or a url, use "current".\nWhen you see a GitLab reference, ResourceIdentifierType should be reference.\n\nMake sure the response is a valid JSON. The answer should be just the JSON without any other commentary!\nReferences in the given question to the current issue can be also for example "this issue" or "that issue",\nreferencing the issue that the user currently sees.\nQuestion: (the user question)\nResponse (follow the exact JSON response):\n```json\n{\n  "ResourceIdentifierType": <ResourceIdentifierType>\n  "ResourceIdentifier": <ResourceIdentifier>\n}\n```\n\nExamples of issue reference identifier:\n\nQuestion: The user question or request may include https://some.host.name/some/long/path/-/issues/410692\nResponse:\n```json\n{\n  "ResourceIdentifierType": "url",\n  "ResourceIdentifier": "https://some.host.name/some/long/path/-/issues/410692"\n}\n```\n\nQuestion: the user question or request may include: #12312312\nResponse:\n```json\n{\n  "ResourceIdentifierType": "iid",\n  "ResourceIdentifier": 12312312\n}\n```\n\nQuestion: the user question or request may include long/groups/path#12312312\nResponse:\n```json\n{\n  "ResourceIdentifierType": "reference",\n  "ResourceIdentifier": "long/groups/path#12312312"\n}\n```\n\nQuestion: Summarize the current issue\nResponse:\n```json\n{\n  "ResourceIdentifierType": "current",\n  "ResourceIdentifier": "current"\n}\n```\n\nBegin!\n', 'temperature': 0.1} request_method=POST request_url=URL('https://api.anthropic.com/v1/messages') source=ai_gateway.models.base

Numbered steps to set up and validate the change are strongly suggested.

Edited by Nathan Weinshenker

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • 2 Warnings
    :warning:

    featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart.

    For more information, see:

    :warning: This merge request does not refer to an existing milestone.

    Reviewer roulette

    Category Reviewer Maintainer
    backend @serenafang profile link current availability (UTC-6, 7 hours behind author) @wandering_person profile link current availability (UTC+7, 6 hours ahead of author)

    Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • added 1 commit

    Compare with previous version

  • Nathan Weinshenker changed the description

    changed the description

  • added 1 commit

    • d3417323 - Create a feature flag passing through model in option parameters

    Compare with previous version

  • added 1 commit

    • 3d7a43ce - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 3 commits

    Compare with previous version

  • added 1 commit

    • 93e92da7 - Create feature flag for rollout of claude gitlab-rails

    Compare with previous version

  • added 1 commit

    • 03cad173 - Update claude_3_5_haiku_rollout.yml

    Compare with previous version

  • Nathan Weinshenker marked this merge request as ready

    marked this merge request as ready

  • Nathan Weinshenker changed title from Draft: Create feature flag for rollout of claude gitlab-rails to Introduce Claude 3.5 Haiku in Gitlab-Rails

    changed title from Draft: Create feature flag for rollout of claude gitlab-rails to Introduce Claude 3.5 Haiku in Gitlab-Rails

  • Nathan Weinshenker resolved all threads

    resolved all threads

  • Nathan Weinshenker changed the description

    changed the description

  • requested review from @lesley-r

  • requested review from @evakadlecova

  • Lesley Razzaghian removed review request for @evakadlecova

    removed review request for @evakadlecova

  • Lesley Razzaghian requested review from @evakadlecova and removed review request for @lesley-r

    requested review from @evakadlecova and removed review request for @lesley-r

  • Eva Kadlecová approved this merge request

    approved this merge request

  • added pipelinetier-2 label and removed pipelinetier-1 label

  • Before you set this MR to auto-merge

    This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3. We will trigger a new pipeline for each transition to a higher tier.

    Before you set this MR to auto-merge, please check the following:

    • You are the last maintainer of this merge request
    • The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
    • This pipeline is recent enough (created in the last 8 hours)

    If all the criteria above apply, please set auto-merge for this merge request.

    See pipeline tiers and merging a merge request for more details.

  • requested review from @wandering_person

  • added pipelinetier-3 label and removed pipelinetier-2 label

  • E2E Test Result Summary

    allure-report-publisher generated test report!

    e2e-test-on-gdk: :white_check_mark: test report for 7647a899

    expand test summary
    +------------------------------------------------------------------+
    |                          suites summary                          |
    +-------------+--------+--------+---------+-------+-------+--------+
    |             | passed | failed | skipped | flaky | total | result |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Verify      | 45     | 0      | 2       | 0     | 47    | ✅     |
    | Package     | 24     | 0      | 11      | 0     | 35    | ✅     |
    | Create      | 129    | 0      | 22      | 0     | 151   | ✅     |
    | Data Stores | 33     | 0      | 1       | 0     | 34    | ✅     |
    | Plan        | 76     | 0      | 0       | 0     | 76    | ✅     |
    | Manage      | 1      | 0      | 1       | 0     | 2     | ✅     |
    | Govern      | 75     | 0      | 3       | 0     | 78    | ✅     |
    | Secure      | 4      | 0      | 0       | 0     | 4     | ✅     |
    | Monitor     | 8      | 0      | 0       | 0     | 8     | ✅     |
    | Fulfillment | 2      | 0      | 0       | 0     | 2     | ✅     |
    | Ai-powered  | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Release     | 5      | 0      | 0       | 0     | 5     | ✅     |
    | Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Total       | 404    | 0      | 41      | 0     | 445   | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+

    e2e-test-on-cng: :white_check_mark: test report for 7647a899

    expand test summary
    +------------------------------------------------------------------+
    |                          suites summary                          |
    +-------------+--------+--------+---------+-------+-------+--------+
    |             | passed | failed | skipped | flaky | total | result |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Govern      | 84     | 0      | 9       | 3     | 93    | ✅     |
    | Plan        | 86     | 0      | 8       | 0     | 94    | ✅     |
    | Create      | 139    | 0      | 21      | 0     | 160   | ✅     |
    | Data Stores | 33     | 0      | 10      | 0     | 43    | ✅     |
    | Monitor     | 8      | 0      | 12      | 0     | 20    | ✅     |
    | Verify      | 50     | 0      | 15      | 0     | 65    | ✅     |
    | Manage      | 1      | 0      | 9       | 0     | 10    | ✅     |
    | Package     | 24     | 0      | 14      | 0     | 38    | ✅     |
    | Fulfillment | 2      | 0      | 7       | 0     | 9     | ✅     |
    | Secure      | 1      | 0      | 6       | 0     | 7     | ✅     |
    | ModelOps    | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Release     | 5      | 0      | 1       | 0     | 6     | ✅     |
    | Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
    | Ai-powered  | 0      | 0      | 2       | 0     | 2     | ➖     |
    | Configure   | 0      | 0      | 3       | 0     | 3     | ➖     |
    | Growth      | 0      | 0      | 2       | 0     | 2     | ➖     |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Total       | 435    | 0      | 120     | 3     | 555   | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
  • Michael Becker
  • added 1 commit

    • 7647a899 - Claude 3.5 Haiku model addition gitlab-rails

    Compare with previous version

  • Nathan Weinshenker
  • Michael Becker approved this merge request

    approved this merge request

  • Michael Becker resolved all threads

    resolved all threads

  • Michael Becker enabled automatic add to merge train when checks pass

    enabled automatic add to merge train when checks pass

  • Nathan Weinshenker changed milestone to %17.6

    changed milestone to %17.6

  • Michael Becker added this merge request to the merge train at position 2

    added this merge request to the merge train at position 2

  • Michael Becker mentioned in commit 5a82afb6

    mentioned in commit 5a82afb6

  • added workflowstaging label and removed workflowcanary label

  • Please register or sign in to reply
    Loading