Skip to content

feat: add tool routing eval config

What does this merge request do and why?

This MR closes issue #1455 (closed) to add a tool routing config that support each tool defines its own routing evaluation user prompt and tool input validation rules. Currently the attribute is optional, once routing evaluation integration is finished on CEF, we will make it a required attribute so we have tool quality control in place.

How to set up and validate locally

  1. Checkout this MR
  2. Update your gdk to the latest version and make sure the gitlab repo is on latest master branch
  3. Run the following curl command
  4. The response will has two attributes, tools for all the tool specs in openai tool format, evalDataset contains all the tool routing eval configs

command:

curl -s --location --request GET "$GITLAB_BASE_URL/api/v4/ai/duo_workflows/list_tools?private_token=$PAT" \                ─╯
    --header 'Content-Type: application/json' \
    --data '{}' | jq '.evalDataset[]'

Expected output, currently we only defined 3 routing evaluation examples, so you just see three configs.

{
  "input_rules": null,
  "tool_name": "read_file",
  "user_prompt": "I need to read the content of the `readme.md`"
}
{
  "input_rules": [
    {
      "arg_name": "file_path",
      "value": "./tools/base.py",
      "operator": "equals"
    }
  ],
  "tool_name": "read_file",
  "user_prompt": "Check if class `DuoBaseTool` exists in `./tools/base.py`"
}
{
  "input_rules": [
    {
      "arg_name": "directory",
      "value": ".",
      "operator": "equals"
    }
  ],
  "tool_name": "list_dir",
  "user_prompt": "I need to list the contents of the current directory"
}

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
  • If this change requires executor implementation: verified that issues/MRs exist for both Go executor and Node executor or confirmed that changes are backward-compatible and don't break existing executor functionality.
Edited by Junming Huang

Merge request reports

Loading