Skip to content

Inject `gitlab_model` data into `results/v2` for evaluation

Dylan Bernardi requested to merge inject-gitlab-model-data into main

This MR injects the gitlab_model data collected in the first iteration of data collection into the current production data collected. The MR also includes the manual script created to inject the gitlab_model data into the current production data.

Note that there are two unique differences to the gitlab_model data:

  1. There are some instances where there was no response from the gitlab_model. This is not unique to the gitlab_model as the other models also often return no response. But, in the second iteration of data collection, for all models except the gitlab_model this was taken care of by ensuring a model response done in this MR: Follow-up: Ensure model response capture during... (!11 - merged). This was not done for the gitlab_model because the triton server had already been shut down by the time v2 data was collected.
  2. Each language/prompt combination is ran 10 times duration data collection to generate average_duration. In the first iteration of data collection, this data was not stored. As a result, the gitlab_model won't include a raw_durations array.

cc @jayswain @mray2020 @srayner

Merge request reports