Skip to content

Adds specs to verify that prompts for a feature, for a specific model exists on AIGW before adding support for the model

Based on the finding in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!2132 (comment 2394745119).

I noticed that we were missing llama3 and gpt prompts for features on Duo chat that are already in use. The lack of existence of the prompt for a model family would lead to Duo Chat erroring out when that specific model is in use.

This is easy to miss because:

  • feature changes are added in gitlab-rails
  • but the prompts should exist in AIGW.

Keeping these in sync can often be missed, so it is the test suite that should be responsible for alerting us if they aren't in sync:

Proposed plan:

* clones/downloads the AI gateway repo's main branch.

* and compares the prompts directory, and it's contents with the feature metadata in the file ee/lib/gitlab/ai/feature_settings/feature_metadata.yml here

* And say, if that file says that llama3 is supported for refactor_code, but we don't find a prompt within prompts/definitions/chat/refactor_code/llama3, the spec fails.

  • AIGW publishes a static site using GitLab pages, at say, https://page-site//prompt_directory_structure.json, on every merge to main
  • This JSON will have chat and code-suggestions sub-features listed, in the following format. This will be generated by a python script that studies the directory structure, and includes the names of model families that has a ".yml" or ".yaml" file inside the directories.
{
  "code_suggestions": {
    "completions": [
      "claude_3",
      "claude_3_user",
      "codegemma",
      "codellama",
      "codestral",
      "codestral_system",
      "deepseekcoder",
      "gpt",
      "llama3",
      "mistral",
      "mixtral",
      "system"
    ],
    "generations": [
      "base",
      "claude_3",
      "codegemma",
      "codellama",
      "codestral",
      "deepseekcoder",
      "gpt",
      "llama3",
      "mistral",
      "mixtral"
    ]
  },
  "chat": {
    "explain_code": [
      "amazon_q",
      "base",
      "claude_3",
      "gpt",
      "llama3",
      "mistral",
      "mistral_nemo",
      "mixtral",
      "system",
      "user"
    ],
....
  • A spec in gitlab-rails will consume this JSON and determine whether a specific model family that's set to be available for a sub-feature has it's prompt defined in AIGW or not.
  • For example, if the file ee/lib/gitlab/ai/feature_settings/feature_metadata.yml here says that llama3 is supported for explain_code, but we don't find llama3 within chat[:explain_code] in the above json, the spec fails.

This way, support for a new model family for a feature cannot be merged on the rails side, without the prompts for it being present in AIGW first.

Edited by Manoj M J