Skip to content

The CreateCommit tool fails to edit any branch other than main without explicit instruction

Problem

When doing testing for gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!3480 (merged) locally I noticed that when I work with a repo that has a master as default branch then it seems to always try to generate a commit to main. This fails and the LLM gets very confused because it's given no information about the failure. The LLM just gets sent:

{
  "error": "Error fetching file 'doc/base_api.md': 'content'"
}

But the actual error sent back from the GitLab API is:

{'message': '404 Commit Not Found'}

The reason it gets this unhelpful error is because we raising an exception at https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/51ef3dd19af0113b316d264ecd8352b18957247f/duo_workflow_service/tools/commit.py#L511 when we try to get content from the JSON payload that is actually {'message': '404 Commit Not Found'}.

There are a few issues here:

  1. The LLM does not know what the default branch is to generate a commit. Perhaps this should be added earlier in the context sent for the initial request
  2. The error handling is problematic because we're losing the error from the GitLab API. We're assuming that anything that parses as valid JSON should be a valid response. But our error responses also parse as JSON. We should probably be passing in use_http_response so that we can check if the response was actually succesful.

As I've been investigating a different issue in #574706 (comment 2802932948) it has became apparent that we'd probably be better off implementing this whole tool on the Rails side as a new API. The Rails side can swap out the content for us. This is necessary for files that come close to the 4MiB gRPC limit because the edit calls will always fail for those large files anyway. If we do that then we might not actually need to fix the above problems. The new Rails code to support this will likely be used in a future MCP server endpoint anyway.

Given the above I recommend we just implement a new Rails API. But per #574714 (comment 2805447398) we probably also need short term fixes in the Python side as well.

Solution

Here is my overall proposal:

  1. Update the Python code so that the response of the API call is never returned from the tool call. Only success/failure should be returned. This reduces the context growth and wasted tokens that lead to overflowing any of our limits eventually.
  2. Update the initial user prompts in agentic chat to include a branch name so that the LLM knows which branch it should generate MRs against. Additionally advise the agent to always create new branches and generate an MR rather than committing straight to the default branch. My tests seem to always show it prefers to create a commit straight into main.
  3. Make error handling improvements to the Python tools as shown in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!3529 (comment 2805394691) . All errors should be logged and the LLM should be given clearer error details about what went wrong.
  4. Add support on the Rails side for doing file edits with old_str and new_str rather than having to implement this on the Python side.

Quality Criteria

All MRs should include langsmith traces and screenshots and we can use this information during review to figure out if the approach is suitable. There might be yet more tradeoffs we need to make here and it's easiest to understand those by examining the prompts and UX.

Edited by Dylan Griffith