GitLab duo responses sometimes don't include the code they reference
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
When querying GitLab Duo with certain requests ,that require code in the response, it can sometimes generate an answer that references code it doesn't share.
For example we had a customer (ZD Ticket Link) report this behavior with queries around yarn. For example: "please create a pipeline to check for yarn" "write a simple pipeline for enabling yarn". Both of these requests result in the following responses (images included in screenshot section)
I replicated this customer's experience by asking a similar question. During my testing I asked twice and got similar responses. After about 10 minutes I asked again (the same question) and got a response with the code in the output. (images included in screenshot section)
Steps to reproduce
Using a project/group/instance with duo enabled enter a prompt revolving around creating a pipeline or sample code. In our examples we used yarn as it seems to have a higher failure rate compared to other prompts.
What is the current bug behavior?
When querying GitLab Duo with certain requests ,that require code in the response, it can sometimes generate an answer that references code it doesn't share.
What is the expected correct behavior?
Any responses that include references to code or pipelines should share the code in the response.
Relevant logs and/or screenshots
Customer screenshots:
My testing screenshots
Read internal note on log info*
Results of GitLab environment info
My test instance is 17.2.1 omnibus.
Customer is on 17.3.4 dedicated.