Skip to content

Add function to capture raw_duration responses and write to JSON blocks

Dylan Bernardi requested to merge collect_raw_durations_in_JSON_block into main

Resolves this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/417168#note_1461973815+

This function loops through JSON responses and injects raw_durations into the JSON block. These raw time durations are being used to calculate average_duration for each model/prompt/language response.

Preview of the JSON object:

"anthropic": {
    "result": "def reverse_string(text):\n    return text[::-1]",
    "duration": 0.9694501429912634,
    "result_length": 47,
    "average_duration": 1.0558840074954787,
    "raw_durations": [
      0.9694501429912634,
      1.142317871999694
    ],
    "uuid": "c322524e-f86f-4982-a985-003c5a3927aa"
  },
  "anthropic_fast": {
    "result": "# def reverse_string(text):  \ndef reverse_string(text):\n    return text[::-1]",
    "duration": 1.0137719890044536,
    "result_length": 77,
    "average_duration": 0.7395080913314208,
    "raw_durations": [
      1.0137719890044536,
      0.583216179991723,
      0.6215361049980856
    ],
    "uuid": "0020e4c4-755e-4123-9ef3-2daa31749f61"
  },

cc @srayner @jayswain @allison.browne

Merge request reports