Respect user selected model when changing mid agentic chat

What does this MR do and why?

This MR fixes a bug introduced by the combination of !251007 (merged) and gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#2174.

The combination of allowing users to switch model mid chat (previously a new chat would be opened if you switched model), and storing the model used and reusing it in future requests (intended for default models which load balance between multiple providers, but also cache when on the same provider, therefore, we should only use the load balancing logic on the first request) introduced a bug where it would appear as if the user has switched model, but the chat will always use whichever model was selected in the first prompt.

This MR fixes this logic in 2 places:

  1. ee/app/services/ai/duo_workflows/connection_config_service.rb Only reuse the previous model if
    1. The user has not passed a preference (using default).
    2. The duo_workflow_provider_stickiness flag is enabled.
    3. There is a previous model stored.
    4. The previous model included is in defaults.
  2. ee/app/services/ai/duo_workflows/create_checkpoint_service.rb Update the model_metadata_json on subsequent requests

References

gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#2901 (closed)

How to set up and validate locally

Ideally to be tested with multiple providers (AWS, Anthropic, Vertex). But if you do not have multiple providers set up, confirm switching between models of the same provider.

  1. Enable Feature.enable(:duo_workflow_provider_stickiness)
  2. Tail the logs gdk tail duo-workflow-service | grep "Request to LLM"
    1. This will display Request to LLM which has request_url towards the end and Request to LLM complete which has model_name
  3. Start a chat using default
    1. Confirm a claude-sonnet-4-6 model is used, and is sent to aiplatform.googleapis.com or api.anthropic.com
  4. Confirm Ai::DuoWorkflows::Workflow.last.model_metadata_json has the expected name for the model (claude_sonnet_4_6_vertex for vertex, or claude_sonnet_4_6 Anthropic)
  5. Send a few more messages, and confirm the same endpoint is used (If you have Vertex and Anthropic models to confirm load balancing is not called for the default model).
  6. Switch any other model/provider
  7. Send a new message, and confirm the expected name/endpoint are used
  8. Confirm Ai::DuoWorkflows::Workflow.last.model_metadata_json has updated.

Confirm switching from a model included in defaults, to "default" does not change anything.

If a user is already using claude_sonnet_4_6_vertex vertex and they switch to default, it should still use the vertex model every time, and never switch to Anthropic (and vice versa)

If a user is using any other model that is not included in the defaults, and they switch to default, it should load balance between either of the defaults.

  1. Start a chat using either of the two models we default to

image.png

  1. Switch to default and confirm it continues using same model (through Ai::DuoWorkflows::Workflow.last.model_metadata_json or the tail used above).
  2. Switch to the other default provider and confirm the used model switches.
  3. Switch back to default, and confirm it continues using the model you just selected.
  4. Switch to any other model and confirm switch again.
  5. Switch back to defaults, and it should use either of the two defaults.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Kaveh Nejad

Merge request reports

Loading
Loading