Skip to content

fix: register aggregate suggestions to `register_model_output_length`

What does this merge request do and why?

fix: register aggregate suggestions to register_model_output_length

Currently, each suggestion overwrites the dict attribute, hence only the last suggestion is counted as token in the Prometheus instrumentator. This change aims to fix that by registering the aggregate suggestions to the instrumentator

#486 (closed)

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Checkout this branch
  2. Run a v2 code completions endpoint with choices_count > 1
  3. Check the instrumentator to see that the model_output_length prints out the expected length

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Closes #486 (closed)

Edited by Jeff Park

Merge request reports