Meet user expectation that Duo Chat should know everything the user sees and beyond
### In short
* Why: We get a lot of feedback from users that they feel the chat does not know anything. They expect the chat to be all-knowing about their data in GitLab?
* What we don't know: While we have some user questions from the chat bashes and other feedback, we don't have a clear understanding of the value of these use cases.
* Goal:
1. Make the chat in GitLab more aware of GitLab context to address the user concern, even if we can't test these new capabilities well do to lack of understanding of the use cases.
2. Learn more about these use cases as users use the chat with these new capabilities. (We can expect to get more distinguished feedback once chat has these capabilities.)
3. Inform domain teams about these learning so they can build out the contexts and test cases.
* UX involvement:
* Initially zero. We will build these capabilities in the chat engineering team.
* Once we have built these we will get more useful user feedback and can use UX bashes and other UXR to get a better understanding of the needs.
* Status:
* https://gitlab.com/gitlab-org/gitlab/-/issues/464587+s will be release in 17.3
* https://gitlab.com/gitlab-org/gitlab/-/issues/468460+s in progress
* Next up: https://gitlab.com/gitlab-org/gitlab/-/issues/468461+s
* Once we have those done we will turn to [questions that relate to more than one issue](https://gitlab.com/groups/gitlab-org/-/epics/14085#questions-that-relate-to-more-than-one-resource). We will use the Search API to address this.
### Status update: November 2024
1. With the releases of MR context, Pipeline Run context and Commit context we have reached the critical mass for supporting questions of the category [Questions requiring the context from one page or from one resource](https://gitlab.com/groups/gitlab-org/-/epics/14085#questions-requiring-the-context-from-one-page-or-from-one-resource). We can park adding more contexts in this category for now.
2. Explicit context adding: The work on code related contexts done by the context task force needs user testing (status: started) to be certain it works as expected. Then it can be released. We need to do [some polishing](https://gitlab.com/groups/gitlab-org/-/epics/15832) (after the release). (Note, you won’t find many questions in the above epic for these categories as the user bashes where mostly done in the chat in GitLab not in the IDE. We have done a few IDE related chat bashes but I haven’t weeded through those questions, yet.) The contexts that the context task force has added to the IDE via a UI picker that allows explicitly naming context to be added are:
1. file context
2. MR context
3. Issue context
4. Dependency context
5. Imports context
6. Local git diff context
3. Next category of contexts should be Questions that relate to more than one resource.
1. Some may require semantic search.
2. Some may require creating links to filtered lists.
3. Some may simply require the agent to support pulling multiple tools or a single tool multiple times. An example is having the context of the linked issues or the child issues of an epic.
I have not started planning work on this category, yet. There are too many other things we have to do. The biggest ones being stability and multithreaded conversations and improve conversation experience (i.e. understand previously discussed context and stick to it, or understand context switches).
4. Explicit context adding in chat in GitLab UI: In parallel, we can see how we can bring the new Explicit context adding built for the IDE also to the chat in the GitLab UI. Not all contexts makes sense here as the focus is less on development. But bringing the foundational experience also to the Chat in the GitLab UI is probably valuable.
Finally, the anticipated (and hoped for) learning explained in the problem statement seems to materialize. We learned that users seem to be interested in asking questions about their GitLab CI. The learning has been shared with the CI team for their consideration and prioritization to contribute to the chat: https://gitlab.com/gitlab-org/gitlab/-/issues/498465+
# Problem
**Users naturally assume and expect that Duo Chat knows all the context data in GitLab that the user sees or could see.** This epic collects questions that users have asked in the chat. These do provide some insight into which questions users want to have answered and which use cases they are trying to solve with the chat. However, not in every case it is clear how valuable such use cases would be. To learn more about this we should aim to meet the users' demand of offering an _all-knowing-chat_ even if we can't test the use cases well due to lack of detail knowledge. As the chat offers more context knowledge we will receive more valuable feedback about their uses of the chat.
<details>
<summary>Click to expand a selection of user quotes and statistics around this problem</summary>
* Sometimes it understands the context of where I am in GitLab, but other times it doesn't, and I have no idea when and why. For example, why can it summarize issues but not MRs or code files? My recommendation: Ideally, Chat should have context everywhere I go. But if it doesn't, maybe there's a way to indicate that before I ask a question? (Source: [Heuristic evaluation spreadsheet Duo Chat tab row 20](https://docs.google.com/spreadsheets/d/1GbjbhdxlrrNZ4EW1x3_f0C6XuFd40drtkf_lgrtQIJE/edit#gid=745918560))
* [It can’t answer questions about groups or projects given by name](https://gitlab.com/gitlab-org/gitlab/-/issues/451441#note_1883188268).
* Trying to get information about commits within a few repos. ... I am expecting the Duo to answer above questions with the context from the repo. Its not of any use if it does not have that context. (Source: Chat feedback modal in the product.)
* Hi team, does Duo chaat not have context of the window it's running in? I'm in an MR and was asking it some questions and all of them failed to give me an answer (Source: [internal user feedback in slack](https://gitlab.slack.com/archives/C06D5C70MD2/p1715737868451839))
* A [PubSec customer](https://gitlab.slack.com/archives/C051SLP8WNB/p1716412786640019): ... one missing feature in the Duo Chat Web ~~IDE~~ (and to a lesser extent VS Code) is allowing prompt queries to include more information in the prompt scope such, for example, if your browser is on a gitlab project pipeline config page send the config with the prompt automatically, or, if when you’re reviewing a change send the file under review along with any user. Maybe create a “permissive mode” for the Code Duo Chat.
Statistics:
* Statistically, this problem ranks among the top negative feedbacks we receive through the [feedback mechanism in the product](https://10az.online.tableau.com/#/site/gitlab/views/DuoFeedbackDashboard/FeedbackDashboard?:iid=2).
* In our [latest Chat Bash](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343) half the questions that ranked low (answers rated 1 or 2 on a scale from 1 to 5) did so because of missing context and users expecting that Chat should be aware of it.
</details>
**The following subsections analyze and categorize the **[**different user question found in chat bashes**](https://docs.google.com/spreadsheets/d/1_WcIL41Hgh-dmsvpgkDBneY0VOyBaH2YAKdiijfVvAc/edit#gid=1759850121&range=A1)** that Duo Chat can NOT answer, yet**.
<details>
<summary>Click to expand how this analysis was carried out</summary>
[Chat bashes are user tests that we run regularly](https://gitlab.com/gitlab-org/ux-research/-/issues/2513). User are asked to use Duo Chat and report their questions, Duo's answers, their ratings and how the answer could have been better. They are not given specific instructions of what kind of questions they can or should ask. The reason is that we want the to use it like any user would use it.
That said only 18% of bash question fall into this group of questions that cannot be answered. The bashes are dominated by questions relating to gitlab documentation. That may be related to multiple reasons:
* The bashes were only conducted in the GitLab Web App. So, we don't expect developer questions that relate to better understanding the repo. ([We are planning a bash focused on Chat usage in the IDE](https://gitlab.com/gitlab-org/ux-research/-/issues/2513#note_1929111826).)
* It can also be assumed that when users try these questions and don't get an answer they are not going to keep trying to ask these types of failing questions. That is true in the bash and in reality.
Details to the [categorization are documented in the file that collects all feedback](https://docs.google.com/spreadsheets/d/1_WcIL41Hgh-dmsvpgkDBneY0VOyBaH2YAKdiijfVvAc/edit#gid=0&range=A1).
</details>
## Questions requiring the context from one page or from one resource
### Questions relating to MRs
<details>
<summary>Click to expand how we are addressing this</summary>
Bash questions relating to MRs have been captured here: [New Data Set: Questions on MRs](https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/prompt-library/-/issues/299#questions-from-duo-chat-bashes-and-in-product-feedback)
There is also already a feature planned to address the lack of MR context: https://gitlab.com/gitlab-org/gitlab/-/issues/464587+s
</details>
### Questions relating to other resources
<details>
<summary>Click to expand the questions table</summary>
| No. | Date of Bash | Query | Comment | Deep link |
|-----|--------------|-------|---------|-----------|
| 1 | 7/31/2023 | How can I fix this failed pipeline job? https://gitlab.com/gitlab-org/gitlab/-/pipelines/948876861 | | [J40](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J40) |
| 2 | 7/31/2023 | Summarize this commit. | User comment: I had a public MR open, actively browsing the diff. The bot indicated it needs more information when the UI has all the information requested to complete the desired ask. | [J47](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J47) |
| 3 | 10/30/2023 | Summarise the page | User comment: I was on GitLab org's main page. If Duo can't summarise the page, I expect it to fail instead of giving falsy answers | [K6](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K6) |
| 4 | 10/30/2023 | What is the change in this commit? | User comment: Given I am on the specific commit's page, I expect Duo to be able to generate some insight about the commit and not give a generic reply | [K20](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K20) |
| 5 | 10/30/2023 | What is the commit about https://gitlab.com/gitlab-org/build/CNG/-/commit/0058cc042929d42bf347ce296806610239597d73 ? | | [K22](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K22) |
| 6 | 10/30/2023 | What page I am on? | | [K28](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K28) |
| 7 | 10/30/2023 | What current page do you see? | User comment: Maybe it technically can not "see" the page, the page here is "the context" I am speaking about. If I have to copy/paste stuff from the page to the Duo, it is a no go. | [K29](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K29) |
| 8 | 2/28/2024 | What is this pipeline for?<br><br>What is this pipeline for: | | [S10](https://docs.google.com/spreadsheets/d/1MRv76qFQIjkN_o5hDvJCtG4m5Y8bLo0LBGIms3lxvSk/edit#gid=1357574929&range=S10) |
| 9 | 3/26/2024 | Where is the suggested pipeline config? | | [F43](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F43) |
| 10 | 3/26/2024 | Merge request pipeline #xxxxxxx failed. Can you run a Root Cause Analysis to suggests ways to fix failed jobs? | PM comment: It is desirable that chat does such root cause analysis. In fact, we are working on such a feature https://gitlab.com/groups/gitlab-org/-/epics/13080+. However, such questions must only be anssered when the user has a Duo Enterprise add-on. | [F63](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F63) |
| 11 | 4/17/2024 | can you tell what is this commit about? | | [G3](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G3) |
| 12 | 4/17/2024 | What is most used language in this repository? | User comment: It asked for extra details about repository, but I was already within repository. | [G10](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G10) |
| 13 | 4/17/2024 | What is most used language in repository https://gitlab.com/xxx/yyy/zzz | User comment: Provide better guidance for UI or integrate with APIs to get answer directly. | [G11](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G11) |
| 14 | 4/17/2024 | Explain the CI setup for this repo. | | [G48](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G48) |
| 15 | 4/17/2024 | This is a java service that's built using gradle deployed via a gitops change in argocd. Any suggestions on how to improve this gitlab.ci.yaml file? | | [G49](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G49) |
| 16 | 4/17/2024 | how often is my ci pipeline scheduled to run for my "renovate-runner" project? | | [G76](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G76) |
| 17 | 3/17/2025 | It would be nice to be able to ask Duo Chat about snippets on their new/index/show/edit/ pages.<br><br>Also, it would be interesting to be able to have Duo Chat save a particular code response as a Snippet. | | [internal feedback](https://gitlab.com/gitlab-org/gitlab/-/issues/430124#note_2401532174) |
</details>
<details>
<summary>Brainstorming and actions</summary>
<table>
<tr>
<th>Objective</th>
<th>Possible Technical Approach</th>
<th>Action Items</th>
</tr>
<tr>
<td>Enable Chat to read the context of the current resource where the user is sitting at when it is necessary</td>
<td>Develop a currentResourceReader and provide the LLM instructions on how to use it</td>
<td>
* Build a prototype that can read the current: Commit, Pipeline.
* Modify the prompt to make user of this new tool.
* Develop a test dataset that will check how effective this solution is
* Incorporate the new tool into the Chat for every page it is available.
* Allow the user to specify the Commit and/or Pipeline.
</td>
</tr>
</table>
Notes:
- This would be a somewhat generic solution, but still focused on specific resources that we can test against. It would be a good stepping stone for a more generic one that can truly address any page.
- We may have to discuss with Pipeline Execution team to see if their work will bring the full pipeline context to Chat and possibly have some building blocks in place to solve this problem.
</details>
## Questions that relate to more than one resource
### Questions expecting Lists as output
<details>
<summary>Click to expand the questions table and further details about the nature of these questions</summary>
See also this research on [GLQL JTBD](https://gitlab.com/gitlab-org/plan-stage/knowledge-group/-/wikis/Misc-Pages/Rough-Draft-of-GLQL-JTBD).
The following questions could all be answered either
1. with a link to an existing GitLab page filtered by the requested data
2. with an API call and the answer presented in the chat
Depending on the length of the resulting list of items one or the other may be better.It seems that these questions are somewhat in the domain of learning. User may simply not know where they can find these lists and need help with navigation and filtering the lists.Are follow-up questions likely? Maybe if the list is long and one wants to further refine it??? Maybe not???
| No. | Date of Bash | Query | Comment | Deep Link |
|-----|--------------|-------|---------|-----------|
| | 7/31/2023 | What are the programming languages used in this group? | Could be answered with a link to the language statistics page. | [J3](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J3) |
| | 7/31/2023 | Can I please have a list of all issues that need my attention? | | [J24](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J24) |
| | 7/31/2023 | Can you show me all open issues assigned to me? | | [J33](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J33) |
| | 7/31/2023 | What is the oldest open issue in this project: https://gitlab.com/gitlab-org/gitlab-docs? | | [J37](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J37) |
| | 7/31/2023 | Do I have any Issues assigned to me with a "severity::1" label? | | [J57](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J57) |
| | 7/31/2023 | Is there anything on this board assigned to milestone 16.3 which is currently blocked? (asked while on a milestone issue board) | Not possible today: one cannot filter by blocked on boards | [J46](https://docs.google.com/spreadsheets/d/1mHONKWvfAVPfaCK_PH0FLJzAFocLSXwRay1KDsWuFqU/edit#gid=1777616728&range=J46) |
| | 8/8/2023 | Give me the list of remaining 16.3 open issues? | | [J7](https://docs.google.com/spreadsheets/d/19NGzHY0JLSFrn4sVGmeEwfq-Inh51ISr2TOGQsFEIUo/edit#gid=1777616728&range=J7) |
| | 2/28/2024 | Find the last GitLab project I contributed to? | Note: this may require access to user activity. So, far Duo only has access to group/project resources | [S3](https://docs.google.com/spreadsheets/d/1MRv76qFQIjkN_o5hDvJCtG4m5Y8bLo0LBGIms3lxvSk/edit#gid=1357574929&range=S3) |
| | 2/28/2024 | First question: Which merge requests do I have on this repo?<br>Followup question: Which merge requests do I have on the [gitlab-org/gitlab](https://gitlab.com/gitlab-org/gitlab) repo? | | [S18](https://docs.google.com/spreadsheets/d/1MRv76qFQIjkN_o5hDvJCtG4m5Y8bLo0LBGIms3lxvSk/edit#gid=1357574929&range=S18) |
| | 3/26/2024 | How do I show jobs which have a status of either failed or canceled? | | [F39](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F39) |
| | 3/26/2024 | Show me what changed in the last merge into develop for this repo. | Note: the ideal answer here would probably be the link to the MR but one could also link to the MR list filtered by the branch and then sorted by latest at the top. | [F67](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F67) |
| | 4/17/2024 | What issues are open on my projects? | | [G72](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G72) |
| | 4/17/2024 | What MRs do I have open? | | [G47](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G47) |
| | 4/17/2024 | What are the open merge requests on the [gitlab-org/gitlab](https://gitlab.com/gitlab-org/gitlab) repo? | | [G89](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G89) |
| | 4/17/2024 | What is the latest successful pipeline of a release/xyz branch for project https://gitlab.com/aaa/bbb/ccc? | | [G5](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G5) |
| | 4/17/2024 | What environments are running in this repo? | | [G50](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G50) |
| | 4/17/2024 | Where can I see how many commits I have committed? | | [G71](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G71) |
| | 4/17/2024 | Can you provide statistics about repositories in perun group? | User comment: What be really cool if the chat is integrated with gitlab APIs and provide data about specific repository / group / user etc.<br>PM comment: this answer could also be answered with a link to the group repo statistics page. But maybe the API request is more powerful in this case. | [G9](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G9) |
| | 5/22/2025 | User asked on the MR list page: Can you chart the data on this page by date? | Feature request: Introduce context of Work Item list pages to Duo Chat. | https://gitlab.com/gitlab-org/gitlab/-/issues/537017+ |
</details>
### Questions relating to the code somewhere in the repo (i.e. not the current file)
Some research around this: [Duo Context Research Insights](https://docs.google.com/presentation/d/1yEfl-TwphKiV3UUWjbNMl2Aq_VD1ehH77rIImcFVCEU/edit#slide=id.g12b319f6181_0_0)
<details>
<summary>Click to expand the questions table</summary>
| No. | Date of Bash | Query | Comment | Deep Link |
|-----|--------------|-------|---------|-----------|
| | 9/12/2023 | Where is BYPASS_SCHEMA_VERSION mentioned in GitLab? | User comment: I expect the chatbot to be able to search in GitLab code base (OSS version) | [K9](https://docs.google.com/spreadsheets/d/1OCn3-RAuxXQBT_HS1-89HUZcDk-FhCzXronyBHuE8VA/edit#gid=1777616728&range=K9) |
| | 3/26/2024 | Show me what changed in the last merge into develop for this repo. | | [F67](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F67) |
| | 4/17/2024 | what is the spring boot version of this project?<br>Is there a newer version for that? | | [G4](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G4) |
| | 4/17/2024 | I am wanting to integrate k6 testing as an automated pipeline feature for this project. How can I best automate that and can you help me get it put into my .gitlab-ci.yml file? | | [G58](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G58) |
| | 4/17/2024 | can you tell me why i keep getting the "Webhook Disabled: A Webook in this group was automatically disabled after being retried multiple times" and how I can fix what is broken? | | [G59](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G59) |
| | 3/26/2024 | Do you have the capability to suggest commit messages given the code change? | PM: this is a question relating to code changes not commited, yet. | [F65](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F65) |
| | 9/10/2024 | Customer: I would like if chat and CS would have a better understanding of the yaml file dependencies. They explained that the pipeline editor already does this and stitch together the different pieces of the ci config files. They would like that CS and chat would be aware of this so they can work in their IDE and get help from Duo on editing/enhancing these files. | | [comment](https://gitlab.com/groups/gitlab-org/editor-extensions/-/work_items/58#note_2101706379) |
</details>
### Questions that require semantic search
Some research around this: [Duo Context Research Insights](https://docs.google.com/presentation/d/1yEfl-TwphKiV3UUWjbNMl2Aq_VD1ehH77rIImcFVCEU/edit#slide=id.g12b319f6181_0_0)
<details>
<summary>Click to expand the questions table</summary>
| No. | Date of Bash | Query | Deep Link |
|-----|--------------|-------|-----------|
| 1 | 8/24/2023 | Tell me what this project is used for. | [K41](https://docs.google.com/spreadsheets/d/1jwQklnXPzbEIodXishsQ1eEjuxcvSRxczx1JrVBZhBo/edit#gid=1777616728&range=K41) |
| 2 | 9/12/2023 | I'm a frontend engineer at GitLab, in the Utilization group, what issue should I work on next? | [K16](https://docs.google.com/spreadsheets/d/1OCn3-RAuxXQBT_HS1-89HUZcDk-FhCzXronyBHuE8VA/edit#gid=1777616728&range=K16) |
| 3 | 9/12/2023 | I'm a frontend engineer at GitLab, in the Utilization group, what issue should I work on next? Give me a frontend issue from gitlab-org/gitlab repository | [K17](https://docs.google.com/spreadsheets/d/1OCn3-RAuxXQBT_HS1-89HUZcDk-FhCzXronyBHuE8VA/edit#gid=1777616728&range=K17) |
| 4 | 10/30/2023 | Please give me a summary of the top issues in this project. | [K9](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K9) |
| 5 | 01/07/2025 | paraphrasing the question: Please, summarize all quarterly retro issues. | https://gitlab.com/gitlab-org/gitlab/-/issues/430124#note_2285026907 |
| 6 | 01/14/2025 (not a bash) | Not specifically a question but a problem: I want to do my talent assessment self evaluation. This needs input from sources in the handbook and could also benefit from directly looking into issues, MRs, etc. that the user has contributed to. (Admittedly a broad problem.) | https://gitlab.com/gitlab-com/office-of-the-ceo/at-at-gitlab/ai-at-gitlab-usecases/-/issues/39+ |
| 7 | 01/20/2025 (not a bash) | Not specifically a question but a problem: I don't know what to work on next. Find issues for me to work on. | https://gitlab.com/gitlab-com/office-of-the-ceo/at-at-gitlab/ai-at-gitlab-usecases/-/issues/41+ |
The following might benefit of semantic search if there is not already a good project landing page that the AI can simply summarize.
| No. | Date of Bash | Query | Comment | Deep Link |
|-----|--------------|-------|---------|-----------|
| 1 | 10/30/2023<br>2/28/2024 | What is this project about? (Question without link and with link to the project.)<br>Explain this project | One user comment: When I was on the project overview page, I expected the answer to the question, to summarize the project page I am on. | [K8](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K8) / [K11](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K11) / [S5](https://docs.google.com/spreadsheets/d/1MRv76qFQIjkN_o5hDvJCtG4m5Y8bLo0LBGIms3lxvSk/edit#gid=1357574929&range=S5) |
| 2 | 2/28/2024 | Please explain what this repo is about. | | [K38](https://docs.google.com/spreadsheets/d/1U2vY33R-vsqQ2ArCh8ts5rA48He4OwH7oqYp9M-8YOo/edit#gid=1777616728&range=K38) |
</details>
### Questions that require contexts from different places (that may not even be in GL)
| No. | Date of Bash | Query | Comment | Deep Link |
|-----|--------------|-------|---------|-----------|
| 1 | 01/23/2025 (not a bash) | Not specifically a question but a problem: I want to create training material or a course. I need to input different documentation as context to interactively create the course material with the AI | | https://gitlab.com/gitlab-com/office-of-the-ceo/at-at-gitlab/ai-at-gitlab-usecases/-/issues/44+ |
| | 2/5/2025 (no bash; question shared in slack) | Can you summarize the commit history of this repository? Can you summarize the commit history of this repository: <url> | | https://gitlab.slack.com/archives/C05GK6M7FBQ/p1738767591667469 |
## Questions requiring deep system knowledge
These questions may be more suited to be solved with Duo Workflows
<details>
<summary>Click to expand the questions table</summary>
| No. | Date of Bash | Query | Chat response | How could Duo Chat's response been improved? | Deep Link |
|-----|--------------|-------|---------------|----------------------------------------------|-----------|
| 1 | 3/26/2024 | how can I make build process faster for the current Gitlab project I'm on? | I have provided a sample optimized .gitlab-ci.yml configuration focused on speeding up builds by leveraging caching, parallel jobs, minimal containers, and separate pipeline stages. Please let me know if you need any other suggestions or clarification around optimizing your builds. | As long as Duo has access to CI editor data I expected it to make suggestions on the specifc CI pipeline. And even a sample configuration was not provided. | [F31](https://docs.google.com/spreadsheets/d/1yR_RxcminS34pIi8O6aTzaa5NAqget6PZtm_OdHQhbo/edit#gid=80954471&range=F31) |
| 2 | 4/17/2024 | can you tell me what is the longest running portion of my CI pipeline for this repo interact-api? | Unfortunately, I do not have access to details about your specific CI pipelines or repositories to determine the longest running portion. As an AI assistant, I can provide general GitLab and DevOps best practices, but require access to pipeline configuration and job logs to analyze performance.<br><br>Since I cannot directly access pipeline data for your interact-api repository, I would suggest checking the CI/CD Analytics page for the project in GitLab. This provides insights into pipeline and job durations, allowing you to identify bottlenecks. You can also review the job logs for each pipeline to see exact timings.<br><br>If you'd like additional help optimizing pipelines or have more details to share, please feel free to provide them and I'm happy to assist further. But without access to repository/pipeline data, I cannot directly determine the longest running job for you. Please let me know if you have any other questions! | It have context awareness. | [G70](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G70) |
| 3 | 4/17/2024 | Why did my runner spend so much time queuing? | Based on the documentation, it seems the runner spent a long time queuing because there was no visibility into expected queue wait times. The documentation mentions that users want metrics to evaluate runner availability and pipeline wait times in order to determine if more runners are needed. Specifically, it states "What is the estimated time in the Runner queue before a job can start?" as an important question.<br><br>However, the documentation does not provide a clear answer as to why this specific runner spent so much time queuing. It discusses queue times and metrics in a general sense, but does not analyze the root cause for this particular long queue time.<br><br>So in summary, while metrics and visibility into queue times are important for managing runners, the documentation does not provide enough details to diagnose the specific long queue time mentioned in the original question.<br><br>Sources:<br>CI Builds and Runner Fleet metrics database architecture | It's a word salad. Not sure what it's trying to tell me.<br><br>Link to the doc is helpful, but the paragraph isn't. | [G74](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G74) |
| 4 | 4/17/2024 | For my entire organization, Rarefied Atmosphere, can you summarize or tell me about any of my DORA metrics for any/all of my projects? | I apologize, but as an AI assistant without access to private organizational data, I cannot provide any summary or details about DORA metrics for the Rarefied Atmosphere organization's projects. I can only access publicly available GitLab documentation and features. If you have any general questions about DORA metrics or configuring them in GitLab, I would be happy to assist based on the information I have access to. However, without access to private project/organizational data, I cannot directly summarize or tell you about custom metrics. Please feel free to provide any public documentation links or further details if you think it could help me improve my response. | It needs context to be useful--it has to be context-aware to offer me more than buying a Claude Opus or ChatGPT subscription for my engineers. | [G80](https://docs.google.com/spreadsheets/d/1as7xnE4Vdu8eHIZivHa_BfWiY-jd8waAAHR07-5QP9A/edit#gid=2085813343&range=G80) |
| 5 | [Feedback](https://gitlab.com/gitlab-org/gitlab/-/issues/430124#note_1934974110) on 6/4/2024 | Case 1: Summarizing Progress From OKR/Issues<br>Case 2: Summarizing Updates From Issue Board<br>Case 3: Compiling Final Test Platform Level Update | | | [Feedback](https://gitlab.com/gitlab-org/gitlab/-/issues/430124#note_1934974110) |
| 6 | [Customer Feedback (see line \`#10\`)](https://gitlab.com/gitlab-com/account-management/eastern-north-america/nasdaq/-/issues/91 "GitLab Duo Pilot - Open Mic Questions ( Master Tracker)") on 6/20/2024 | It is not fully clear what the use case or the question is. It is being clarified [here](https://gitlab.com/gitlab-com/account-management/eastern-north-america/nasdaq/-/issues/91#note_2025644828 "GitLab Duo Pilot - Open Mic Questions ( Master Tracker)"). | | | [Customer Feedback (see line \`#10\`)](https://gitlab.com/gitlab-com/account-management/eastern-north-america/nasdaq/-/issues/91 "GitLab Duo Pilot - Open Mic Questions ( Master Tracker)") |
| 7 | [In-product user feedback](https://docs.google.com/spreadsheets/d/1UngqjcnrwqhaykpVt1B8G_RWKXSp3uGzIWNQi1_2d3M/edit?gid=821072114#gid=821072114&range=C5) | Not a query but a goal: Asking it a question about the issues in a pipeline whilst being on the pipeline page. It appears to have no ability to pull context from the page I'm currently looking at. The yml config for this particular pipeline is extremely complex so having the AI bot be able to pull context from the page I'm currently viewing would be extremely handy. | | | |
</details>
# Solution proposal
After a [team discussion and team brain storming](https://gitlab.com/groups/gitlab-org/-/epics/14085#note_1943473110) we decided
- we will start adding a few additional contexts that relate only to one page or from one resource
- in parallel, we will consider if there is a path to enable all such contexts in GitLab with one solution
- we will also consider working on use cases that require leverage global search for RAG.
#### Legal Considerations
* https://gitlab.com/gitlab-org/gitlab/-/issues/488888#note_2103464280+s
epic