Skip to content

Migrate Code Generation Prompts: Mistral-7B-v0.1

This issue is the pilot / PoC for all of the prompt migration issues in %17.3

Migration of the Code Suggestions Prompts for Mistral-7B-v0.1 to the AI Gateway.

  • Behind the :ai_custom_model feature flag

See the parent Epic Prompt Migration to AI Gateway (&14259 - closed) for links to the original prompt code.

Definition of Done

  • Update Rails code to point to AI Gateway prompt
  • Add prompt to AI Gateway
  • PoC Prompt evaluation using LangSmith
  • Full evaluation with CEF

Using LangSmith via ELI5

For "PoC Prompt evaluation using LangSmith", please use ELI5. This is a project that simplifies the process of running evaluations using LangSmith. To use LangSmith via ELI5 for code suggestions evaluation as an example:

poetry run eli5 code-suggestions evaluate \
  --dataset="code-suggestions-input-testcases-v1" \
  --source=gitlab \
  --offset=100 \
  --limit=10 \
  --experiment-prefix=experiment-123

The evaluation results will be displayed in the console and a link to view detailed results in LangSmith will be provided.

Edited by Sean Carroll