Skip to content
Snippets Groups Projects
Verified Commit f2b33c59 authored by Amy Qualls's avatar Amy Qualls :leopard: Committed by GitLab
Browse files

Merge branch 'selhorn-move-eval-runner' into 'master'

Moving eval runner docs

See merge request !179907



Merged-by: default avatarAmy Qualls <aqualls@gitlab.com>
Approved-by: David O'Regan's avatarDavid O'Regan <doregan@gitlab.com>
Approved-by: default avatarAmy Qualls <aqualls@gitlab.com>
Reviewed-by: default avatarShinya Maeda <shinya@gitlab.com>
Reviewed-by: default avatarManoj M J <mmj@gitlab.com>
Co-authored-by: default avatarSuzanne Selhorn <sselhorn@gitlab.com>
parents a74c8f22 fdce89c5
No related branches found
No related tags found
3 merge requests!181325Fix ambiguous `created_at` in project.rb,!180727Resolve "Extend job archival mechanism to the whole pipeline",!179907Moving eval runner docs
Pipeline #1668244513 passed
---
stage: AI-powered
group: AI Framework
info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review.
title: Evaluation runner
---
Evaluation runner (`evaluation-runner`) allows GitLab employees to run evaluations on specific GitLab AI features with one click.
- You can run the evaluation on GitLab.com and GitLab-supported self-hosted models.
- To view the AI features that are currently supported, see
[Evaluation pipelines](https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner#evaluation-pipelines).
Evaluation runner spins up a new GDK instance on a remote environment, runs an evaluation, and reports the result.
For more details, view the
[`evaluation-runner` repository](https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner).
## Architecture
```mermaid
flowchart LR
subgraph EV["Evaluators"]
PL(["PromptLibrary/ELI5"])
DSIN(["Input Dataset"])
end
subgraph ER["EvaluationRunner"]
CI["CI/CD pipelines"]
subgraph GDKS["Remote GDKs"]
subgraph GDKM["GDK-master"]
bl1["Duo features on master branch"]
fi1["fixtures (Issue,MR,etc)"]
end
subgraph GDKF["GDK-feature"]
bl2["Duo features on feature branch"]
fi2["fixtures (Issue,MR,etc)"]
end
end
end
subgraph MR["MergeRequests"]
GRMR["GitLab-Rails MR"]
GRAI["AI Gateway MR"]
end
MR -- [1] trigger --- CI
CI -- [2] spins up --- GDKS
PL -- [3] get responses and evaluate --- GDKS
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment