Skip to content

Inject `prompt_type` into header of current/future evaluation data

Dylan Bernardi requested to merge inject-prompt-type into main

This MR includes two parts:

  1. The evaluation script that is used to collect model response data is updated to include prompt_type (currently either "generation" or "completion") in the header. This is being added as a result of realizing the convenience of having the data field during analysis.

  2. The current production data is updated to have prompt-type included in the headers.

Validating new header

A test run of generating new prompt results to prove that the header has the prompt_type field in the JSON object response. image

cc @jayswain @srayner

Merge request reports