Skip to content

Resolve "Add used tools to completions rake task output file"

What does this MR do and why?

This MR updates the Duo Chat completions Rake task to include the tool used in the tasks' output. More detailed information can be found in Issue #443676 (closed)

The duo_chat::completions.rake tasks output is obtained from the output of the Gitlab::Duo::Chat::Request.completion method. Which consists of the following format:

{
  ref: data_row.ref,
  query: query,
  resource: resource,
  response: response.response_body
}

The idea behind this MR is to allow for the tasks to include the tool_used within the output so we can ensure our IssueReader and EpicReader tools are being called at the proper times.

Closes #443676 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. you need to have root group path set up - it needs to be on ultimate or premium plan (so you need to run your instance as SaaS) and it needs to have experiment and beta setting switched on
  2. you need to clone dataset repo: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/datasets
  3. into the root group, you need to import data: -zip archive with samples - https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/datasets/-/packages/22409459
    • Zip archive:
      • unzip manually, the root folder contains several other archives that can be imported using GitLab UI.
      • Import the groups first, then import the projects
      • You can import the exported groups to any root group, however you need to preserve the canonical names of the exported groups and projects.
      • We use group and project names and iids to match the queries with resources in our dataset.

Then you can run rake task:

AIEF_DATASET=<DATASET_REPO>/duo_chat/v1/jsonl/queries \
AIEF_OUTPUT=duo_chat_completions GITLAB_SIMULATE_SAAS=true bundle exec rake "gitlab:duo_chat:completions[<root group path>, <user id>]"

When looking at the file generated by the rake tasks you should now see that it contains tool_used and its output should either be nil, Gitlab::Llm::Chain::Tools::IssueReader::Executor, or Gitlab::Llm::Chain::Tools::EpicReader::Executor

Changes To Be Made

Edited by Dillon Wheeler

Merge request reports