Skip to content

Add the script to pull the aggregated vote data

Allison Browne requested to merge ab-vote-aggregation-script into main

Pull the vote data from the command line

Move into the eval code viewer directory

cd eval-code-viewer

Ensure the dependencies are installed through yarn.

Run the script to get the aggregate data:

▶ node tasks/get_evaluations.js
{
  data: [
    {
      model: 'anthropic',
      language: 'javascript',
      total_vote_count: '14',
      thumbs_up_count: '6',
      thumbs_down_count: '8',
      prompt_type: generation,
      avg_response_time: 0.9
    },
    {
      model: 'anthropic_fast',
      language: 'javascript',
      total_vote_count: '12',
      thumbs_up_count: '7',
      thumbs_down_count: '5',
      prompt_type: generation,
      avg_response_time: 0.9
    },
    ...
  ],
  message: 'Successfully retrieved aggregated votes'
}

Run the script that fetches the raw data:

node tasks/get_evaluations.js raw
{
  data: [
    ...
    {
      id: 104,
      prompt_result_id: 'c3462496-6912-4fa7-9814-254c5ce37576',
      language: 'javascript',
      model: 'vertex_text_bison',
      option: 'thumbsDown'
    },
    {
      id: 105,
      prompt_result_id: 'be68d7d6-4b73-4082-86f8-99f66df80295',
      language: 'javascript',
      model: 'anthropic_fast',
      option: 'thumbsUp'
    },
    ... 107 more items
  ],
  message: 'Successfully retrieved raw votes'
}
Edited by Allison Browne

Merge request reports

Loading