Adds specs to verify that prompts for a feature, for a specific model exists on AIGW before adding support for the model
Based on the finding in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!2132 (comment 2394745119).
I noticed that we were missing llama3 and gpt prompts for features on Duo chat that are already in use. The lack of existence of the prompt for a model family would lead to Duo Chat erroring out when that specific model is in use.
This is easy to miss because:
- feature changes are added in gitlab-rails
- but the prompts should exist in AIGW.
Keeping these in sync can often be missed, so it is the test suite that should be responsible for alerting us if they aren't in sync:
Proposed plan:
* clones/downloads the AI gateway repo's main
branch.
* and compares the prompts
directory, and it's contents with the feature metadata in the file ee/lib/gitlab/ai/feature_settings/feature_metadata.yml
here
* And say, if that file says that llama3
is supported for refactor_code
, but we don't find a prompt within prompts/definitions/chat/refactor_code/llama3
, the spec fails.
- AIGW publishes a static site using GitLab pages, at say,
https://page-site//prompt_directory_structure.json
, on every merge tomain
- This JSON will have
chat
andcode-suggestions
sub-features listed, in the following format. This will be generated by a python script that studies the directory structure, and includes the names of model families that has a ".yml" or ".yaml" file inside the directories.
{
"code_suggestions": {
"completions": [
"claude_3",
"claude_3_user",
"codegemma",
"codellama",
"codestral",
"codestral_system",
"deepseekcoder",
"gpt",
"llama3",
"mistral",
"mixtral",
"system"
],
"generations": [
"base",
"claude_3",
"codegemma",
"codellama",
"codestral",
"deepseekcoder",
"gpt",
"llama3",
"mistral",
"mixtral"
]
},
"chat": {
"explain_code": [
"amazon_q",
"base",
"claude_3",
"gpt",
"llama3",
"mistral",
"mistral_nemo",
"mixtral",
"system",
"user"
],
....
- A spec in gitlab-rails will consume this JSON and determine whether a specific model family that's set to be available for a sub-feature has it's prompt defined in AIGW or not.
- For example, if the file
ee/lib/gitlab/ai/feature_settings/feature_metadata.yml
here says thatllama3
is supported forexplain_code
, but we don't findllama3
withinchat[:explain_code]
in the above json, the spec fails.
This way, support for a new model family for a feature cannot be merged on the rails side, without the prompts for it being present in AIGW first.