Skip to content

Combine resource reader and identifier in one tool per resource

Gosia Ksionek requested to merge mk-chat-experiment into master

What does this MR do and why?

Goal

The aim of this pull request is to enhance the speed of our chat system when dealing with questions related to epics and issues.

Current Process

Up until now, answering questions about epics or issues involved using two separate tools:

  1. Issue/Epic Identifier: This tool helped identify the specific issue or epic.
  2. Resource Reader: The Resource Reader tool was used to serialize the resource into a format suitable for the large language model. Previously, this format was JSON, but it has now been changed to XML.

Previous Workflow

The previous workflow involved multiple back-and-forth interactions with the large language model:

  1. First, we would select the appropriate tool, either the issue or epic identifier.
  2. Then, we would query the large language model with the results from the identifier to determine which resource reader to use.
  3. Next, we would pass the identified issue or epic to the resource reader for serialization.
  4. Finally, we would feed the serialized issue or epic back into the large language model and wait for the final answer.

Proposed Workflow

With the changes introduced in this MR, we've streamlined the process by eliminating the middle step:

  1. Upon receiving the initial question from the customer, we immediately consult the large language model.
  2. The model indicates that we should use the epics/issue reader.
  3. The epics/issue reader identifies the resource (epic or issue) and serializes it instantly.
  4. We then input this serialized data directly into the large language model, which provides us with the final answer.

Pros and Cons

Pros: The primary advantage of this solution is its speed improvement. It significantly reduces the time taken to respond to queries about epics and issues.

Cons: One drawback is that when we initially designed the chat system, we anticipated having more tools related to epics and issues. The issue or epic identifier was intended to be a common tool for such tasks. Additionally, the resource reader could have been utilized for other models, such as merge requests. However, the current lack of additional tools and the observed latency in the chat system prompted us to prioritize speed over versatility. The tools we used previously are still in the codebase, we can start using them at any time.

  • it's worth noting that by merging these tools, we're disregarding a specific use case present in the resource reader. This use case was originally designed to handle scenarios where resources were deeply nested in JSON hierarchies. However, given the substantial token window size in our current tools, this functionality is no longer relevant and can be considered dead code.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Recording comparing speed: internal only: https://drive.google.com/file/d/1Q7XagR3SbDiemY5yCgBvmmDdJdaHzn39/view?usp=drive_link

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Ask question about epic or issue using reference to the resource (not "this issue" or "current epic"). Observe speed.

Edited by Gosia Ksionek

Merge request reports