Skip to content

Optimize DAP for GPT OSS 120b

Overview

At GPT OSS for DAP (#569773 - closed) we've added support and got evaluation numbers for the current setting of the DAP agent using GPT OSS 120b.

header header
swe-bench 18/31 completed, 0/31 resolved
issue-to-mr 1st run: 2/6 MRs created (although the commits are not very useful), 2nd and 3rd runs: 0/6 MRs created (extremely flaky due to disconnections)

This issue will capture improvements to the agent, such as making tool calls and prompts more specific to GPT as currently they are optimized for Claude models.

Improvements

From the top of my head, some possible low hanging fruits to look at:

  • Fix and optimize tool descriptions and tool calling. The agent is struggling to create merge requests for example #569773 (comment 2857191488).
  • Review the system prompt to work better with GPT.

Other changes:

  • GPT has smaller context-window than Claude, review what can be prioritized and pruned from the conversation history.
  • The agent tends to get stuck in a loop when trying to find more instances of a certain symbol by guessing file patterns. This can already be seen in the simplest example of issue-to-mr, where it keeps trying to find other instances of the renamed class.
  • Review the planner implementation cc @achueshev.
Edited by Bruno Cardoso