Migrate Code Generation Prompts: Mistral-7B-v0.1
This issue is the pilot / PoC for all of the prompt migration issues in %17.3
Migration of the Code Suggestions Prompts for Mistral-7B-v0.1 to the AI Gateway.
- Behind the
:ai_custom_model
feature flag
See the parent Epic Prompt Migration to AI Gateway (&14259 - closed) for links to the original prompt code.
Definition of Done
-
Update Rails code to point to AI Gateway prompt -
Add prompt to AI Gateway -
PoC Prompt evaluation using LangSmith -
Full evaluation with CEF
Using LangSmith via ELI5
For "PoC Prompt evaluation using LangSmith", please use ELI5. This is a project that simplifies the process of running evaluations using LangSmith. To use LangSmith via ELI5 for code suggestions evaluation as an example:
poetry run eli5 code-suggestions evaluate \
--dataset="code-suggestions-input-testcases-v1" \
--source=gitlab \
--offset=100 \
--limit=10 \
--experiment-prefix=experiment-123
The evaluation results will be displayed in the console and a link to view detailed results in LangSmith will be provided.
Edited by Sean Carroll