Baseline Mistral Series for Code Completion
This issue is to capture work and results around the baseline validation assessment for all currently non-supported Mistral models for code completion, to determine if they are also viable for code completion use.
For each variant, we will need to host it in the local GDK and run against the complete CEF datasets for Code Suggestions for Code Completion (dataset_v2) to establish baselines for performance as outlined here.
The model to baseline are:
- mistralai/Mistral-7B-v0.3
- mistralai/Mistral-7B-Instruct-v0.3
- mistralai/Mixtral-8x7B-v0.1
- mistralai/Mixtral-8x7B-Instruct-v0.1
- mistralai/Mixtral-8x22B-v0.1
- mistralai/Mixtral-8x22B-Instruct-v0.1
Note: Codestral 22B is already baselined and supported for Code Completion
Definition of Done
Each Models performance is documented in this issue, providing baseline performance scores in terms of cosine similarity to ground truth.
Edited by Susie Bitters