Skip to content

Score pipeline

Hongtao Yang requested to merge score-pipeline into main

This MR adds the dataflow pipeline for scoring the model response. It will take a model's response, and compare it with the expected output written by a real developer. The higher the score, the more similar they are.

We can assume that higher similarity score means higher output quality.

I've test-run the pipeline, and the results are stored in this BQ table: https://console.cloud.google.com/bigquery?project=unreview-poc-390200e5&ws=!1m5!1m4!4m3!1sunreview-poc-390200e5!2sgl_gitlab_codebase!3sscore_chunks_v2

Ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/5

Edited by Hongtao Yang

Merge request reports