Git command execution fails on staging.gitlab.com

Summary

Git commands that attempt to reach the remote repository are failing on staging with a 403 error barring the Duo Developer and similar flows from pushing their changes. The error slightly changes based on whether Network isolation via SRT is turned on. It changes again / was able to recover on Friday by setting the remote URL manually.

Steps to reproduce

1. Set up a group/project in staging and enable foundational flows (or use https://staging.gitlab.com/dap-flows-test)
2. Set up an issue or use https://staging.gitlab.com/dap-flows-test/dap-flows-1/-/work_items/1
3. Press the "Generate MR with Duo" button
4. Go into the created session to see reports of git tools failing
5. Go to the linked Job to see more clear 403 errors.

Examples with Logs

Error in the job logs:

With Network Isolation (Job)

Error: git command failed with exit code 128: fatal: unable to access 'https://staging.gitlab.com//dap-flows-test/dap-flows-1.git/': Received HTTP code 403 from proxy after CONNECT

Without Network Isolation (Job)

Error: git command failed with exit code 128: fatal: unable to access 'https://staging.gitlab.com//dap-flows-test/dap-flows-1.git/': Could not resolve host: staging.gitlab.comhttps

After agent decided to run git remote set-url origin https://staging.gitlab.com/dap-flows-test/dap-flows-1.git/ :

Error: git command failed with exit code 128: remote: You are not allowed to push code to this project.
    fatal: unable to access 'https://staging.gitlab.com/dap-flows-test/dap-flows-1.git/': The requested URL returned error: 403

On Friday the flow was able to recover via

• Session that did recover
  • Session
  • Job
  • LangSMith
• Session that did not recover
  • Session
  • Job
  • Langsmith

State of Investigation

1. The last error after git remote set-url is the same one as was fixed for Self-Managed by !217964 (merged), pointing to an auth issue, potentially this could be completely fixed now since !217964 (merged) was not merged the last time the agent decided to use this approach
2. The URL in the errors with Network Isolation contains a double slash // after the staging.gitlab.com, and the error without Network isolation also points to an issue with the url since it is trying to resolve a host of staging.gitlab.comhttps
3. We had problems with malformed urls specifically for staging/sub-domains in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!2585 (merged)

Next steps

1. Change Duo Developer to experimental in staging and make it run git remote set-url origin to see whether this can solve or at least change the error.
2. Investigate the Language Server to understand what can cause the double slash. Investigation issue created by Duo: gitlab-org/editor-extensions/gitlab-lsp#1897