Split tool selection and question answering for prompt template
🔦
Objective The Objective is to experiment with Duo Chat to reach par with the foundational model Claude in terms of quality, as measured by the similarity score. Below is the screenshot from the current dashboard overview, that demonstrates where Duo Chat lies in respect to Claude.
#⃣
Primary Metric for Success The primary metric for success in this iteration of experimentation is the Comparison Similarity Score. This score specifically compares the output generated by the Answering Model (Duo Chat) and the Comparison Model (Claude).
📚
Dataset for Diagnostic Testing/Experimentation For this dataset for experiments this is a small subset of data that can be found here as a subset of the dashboard : https://docs.google.com/spreadsheets/d/1zuDH8BGVPb5rtHOogmiAqH3kgEXYD8h3UfHVMfx_SwE/edit#gid=0 which is a subset of 45 Code Generation Question
We have created 3 videos to walkthrough the idea behind this change and an example how to incorporate it to Duo Chat development
- How to run end to end experiments as an example of trimmed prompts (https://www.youtube.com/watch?v=H2oykA5THac)
- Curious on how the datasets are build and run the experiments (https://www.youtube.com/watch?v=swN2EtAzdWA)
- A generic walkthrough of end to end pipeline to get familiar with accessing Prompt Library( https://youtu.be/U2CW95yylMs)
🔍
Metrics - Control Metric Score: Comparison Similarity: (avg similarity score) 0.57
- Experiment Metric Score: TBD post Experiment
✍🏼
Experiment Details Based on prompt-library$3673660, we can see the final prompt that duo-chat constructed is quite noisy.
The experiment we recommend is to split the duo-chat answer process into 2 parts:
-
Tool Selection
Ask a LLM to select a tool based on the question. We can do this by using this prompt template or similar:
Human:
You have access only to the following tools:
1. <tool_description>
<tool_name>CiEditorAssistant</tool_name>
<description>
Useful tool when you need to provide suggestions regarding anything related to ".gitlab-ci.yml" file.
It helps with questions related to deploying code, configuring CI/CD pipelines, defining CI jobs, or environments.
</description>
<example>
Here is an example of using this tool:
Question: Please create a deployment configuration for a node.js application.
Picked tools: "CiEditorAssistant" tool.
Reason: You have asked a question related to deployment of an application or CI/CD pipelines.
"CiEditorAssistant" tool can assist with this kind of questions.
</example>
</tool_description>
2. <tool_description>
<tool_name>ResourceReader</tool_name>
<description>
Useful tool when you need to get information about specific resource that was already identified. Action Input for this tools always starts with: `data`
</description>
<example>
Here is an example of using this tool:
Question: Who is an author of this issue
Picked tools: First: "IssueIdentifier" tool, second: "ResourceReader" tool.
Reason: You have access to the same resources as user who asks a question.
Once the resource is identified, you should use "ResourceReader" tool to fetch relevant information
about the resource. Based on this information you can present final answer.
</example>
</tool_description>
3. <tool_description>
<tool_name>IssueIdentifier</tool_name>
<description>
Useful tool when you need to identify a specific issue. Do not use this tool if you have already identified the issue.In this context, word `issue` means core building block in GitLab that enable collaboration, discussions, planning and tracking of work.Action Input for this tool should be the original question or issue identifier.
</description>
<example>
Here is an example of using this tool:
Question: Please identify the author of #issue_identifier issue
Picked tools: First: "IssueIdentifier" tool, second: "ResourceReader" tool.
Reason: You have access to the same resources as user who asks a question.
There is issue identifier in the question, so you need to use "IssueIdentifier" tool.
Once the issue is identified, you should use "ResourceReader" tool to fetch relevant information about the resource. Based on this information you can present final answer.
</example>
</tool_description>
4. <tool_description>
<tool_name>GitlabDocumentation</tool_name>
<description>
This tool is beneficial when you need to answer questions concerning GitLab and its features.
Questions can be about GitLab\'s projects, groups, issues, merge requests,
epics, milestones, labels, CI/CD pipelines, git repositories, and more.
</description>
<example>
Here is an example of using this tool:
Question: How do I set up a new project?
Picked tools: "GitlabDocumentation" tool.
Reason: Question is about inner working of GitLab. "GitlabDocumentation" tool is the right one for
the job.
</example>
</tool_description>
5. <tool_description>
<tool_name>EpicIdentifier</tool_name>
<description>
Useful tool when you need to identify a specific epic. Do not use this tool if you have already identified the epic. In this context, word `epic` means high-level building block in GitLab that encapsulates high-level plans and discussions. Epic can contain multiple issues. Action Input for this tool should be the original question or epic identifier.
</description>
<example>
Here is an example of using this tool:
Question: Please identify the author of &epic_identifier epic
Picked tools: First: "EpicIdentifier" tool, second: "ResourceReader" tool.
Reason: You have access to the same resources as user who asks a question.
There is epic identifier in the question, so you need to use "EpicIdentifier" tool.
Once the epic is identified, you should use "ResourceReader" tool to fetch relevant information
about the resource. Based on this information you can present final answer.
</example>
</tool_description>
Here is the question from the user: {question}.
Which tool would you choose to answer this question? If none of the tools are suitable for the question, just return "No tool".
Assistant:
-
Question Answering
After a tool has been selected, we can reduce the final prompt by using this template or similar:
<tool>
{tool_description}
</tool>
<user_question>
{user_question}
</user_question>
<context>
{context}
</context>
Human:
Please use the above tool to answer the user question based on the context.
Assistant: