This epic tracks the work to bring the [GitLab Duo Chat](https://docs.gitlab.com/ee/user/gitlab_duo_chat.html) system to Beta. This includes the backend where the heavy lifting is done and the WebUI frontend. **While this epic does NOT include the **[**Chat in VS Code and Web IDE**](https://gitlab.com/groups/gitlab-org/-/epics/10522 "Bringing GitLab Chat to WebIDE & VSCode via Gitlab Workflow Extension")** work, it needs to build the backend in such a way that it can support clients other than the WebUI - specifically IDEs**. It must consider that the IDEs may not have the same context as the WebUI. For instance, an IDE may not know the URL of the code file the user is looking at, as the user might have just created it locally. Also the file may have changed on the client. The maturation of the Chat in VS Code and Web IDE is independent of the chat system maturation. :movie_camera: [Video Walk-through of this epic](https://youtu.be/T645Ppm8oQ0) ## Overall Scope Vision: GitLab Duo Chat is a conversational AI assistant that simplifies tasks and reducing context switches to empower users to achieve their DevSecOps goals faster and streamlined. Beta scope: Enable users to ask questions relating to an issue, an epic, or code, such as "Could you summarize this issue?". Enable users to ask for help to learn how to use GitLab, such as "How do I set up my CI/CD pipeline for my project?". Enable users to ask follow-up questions. ### Functional scope * [x] Chat can be opened anywhere on the GitLab Web page. (Excluding [Web IDE](https://gitlab.com/groups/gitlab-org/-/epics/10522 "Bringing GitLab Chat to WebIDE & VSCode via Gitlab Workflow Extension")) * [ ] Chat answers questions relating to * [x] code the user sees (explain "this") * [x] code referenced (with URL) * [x] issue the user sees * [x] issue referenced * [x] epic the user sees * [x] epic referenced * [ ] ~~MR the user sees~~ (removed from beta scope) * [ ] ~~MR referenced~~ (removed from beta scope) * [x] how to use GitLab * [x] recent context from chat history (i.e. Conversational Chat enabled) * [ ] ~~selections (if something is selected e.g. code in the code file, this is added to the context so the chat can answer questions about the selection)~~ (removed from beta scope) * [ ] dummy tools exist that proof the agent can support many more use cases (pipeline context, global issues/epics/code contexts) * [ ] If context is not available, [chat responds with its supported contexts](https://gitlab.com/gitlab-org/gitlab/-/issues/415801 "Self-discoverable capabilities of the chat via responses of the chat itself - e.g. I can't answer this questions but I can answer questions that are about the code you are looking at..."). * [x] User can give feedback on their satisfaction with the answers and these answers are analyzed to inform potential prompt engineering tweaks. ### Non-functional scope * [x] **By design: The tools and the agent(s) must never share context that the user does not have access to.** * [ ] The quality bar for what success means ([see below](https://gitlab.com/groups/gitlab-org/-/epics/10550#what-does-success-mean "GitLab Duo Chat [Beta]")) is defined, tested and passed * [x] [Perceived speed of answers is at least o.k](https://gitlab.com/groups/gitlab-org/-/epics/10683). from user perspective and there are ideas how to improve it to good. * [x] https://gitlab.com/gitlab-org/gitlab/-/issues/419531+s * [x] https://gitlab.com/gitlab-org/gitlab/-/issues/419531+s * [ ] Errors are tracked. One of these two options. Still TBD: * [ ] Works in the main product for Ultimate users that have toggled the right switches in the settings. * [ ] **Access to the Chat is granted if the user has paid the add-on.** The project or the tier of the project's namespace does not matter. [This still needs to be confirmed](https://gitlab.com/gitlab-org/gitlab/-/issues/421295#note_1515265994 "Discuss: Allow the user to use the chat while not being in a GitLab repo"). * [ ] During the pre-GA phase or pre-addon phase, all SaaS users that are at least in one Ultimate tier, do have access to the chat. [This still needs to be confirmed](https://gitlab.com/gitlab-org/gitlab/-/issues/421295#note_1515265994 "Discuss: Allow the user to use the chat while not being in a GitLab repo"). ## What does success mean? Generally * AI answers should be _mostly_ correct, helpful and friendly. * If the AI does not know it should _mostly_ say that is does not know. #### How to measure it? ~~We shall consider the experience Beta-ready once we have achieved the following:~~ We shall consider the experience satisfactory once we have achieved the following: * Users consider \>80% of responses `good` or `excellent` * Users consider \<5% of responses `terrible` (on a 5 point scale: `terrible`, `poor`, `average`, `good`, `excellent`) See [details why we opted for this quality bar](https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/3555#note_1541945461 "KR2: GitLab Duo Chat becomes available in beta, including six “specialized tools” that are also in beta, with a path to general availability in FY24Q4."). [This user satisfaction will be assessed by UXR roughly every two to three weeks](https://gitlab.com/gitlab-org/ux-research/-/issues/2513 "Repeated analysis of user experience with Duo chat as we continuously improve the chat (also considering the outcome of each round of these user tests.)"). Check the [bottom of our AI Features Dashboard](https://app.periscopedata.com/app/gitlab/1137231/Ai-Features) for these metrics and how they progress toward the goal. ## Work items (remaining as of mid August) #### Understand in more detail the problem space and how well we address user needs (mostly UXR and Product analytics) **Outcomes**: More clear understanding of users expectations and which problems they want to solve interactively with the chat. * https://gitlab.com/groups/gitlab-org/-/epics/11120+ * https://gitlab.com/gitlab-data/product-analytics/-/issues/1325+ * https://gitlab.com/gitlab-org/ux-research/-/issues/2513+ * https://gitlab.com/gitlab-org/gitlab/-/issues/415591+ #### Addressing functional gaps & UX issues **Outcomes**: better user experience measurable in better scores in user testing - ~~MR context: https://gitlab.com/gitlab-org/gitlab/-/issues/412430+s~~ (removed from beta scope; see [reason](https://gitlab.com/gitlab-org/gitlab/-/issues/412430#note_1545114583)) - Epic context: https://gitlab.com/gitlab-org/gitlab/-/issues/422091+s - Support selected text: https://gitlab.com/gitlab-org/gitlab/-/issues/422176+s - Streaming: https://gitlab.com/groups/gitlab-org/-/epics/10683+s - Dummy contexts: https://gitlab.com/gitlab-org/gitlab/-/issues/422187+s - Self-discoverable capabilities: https://gitlab.com/gitlab-org/gitlab/-/issues/415801+s - [Collection of UX issues to be addressed](https://gitlab.com/groups/gitlab-org/-/epics/10673 "Chat Integration iteration 1 FE & BE"). Such as: - new conversation - sync chat across clients - onboarding - expiration time for response - Error messages - feedback mechanism - ~~Global entry point for chat should be enabled for beta release but should not be enabled before we reach our quality target.~~ [Global entry point for chat will only be turned on when we reach the quality target defined above independent of the beta release](https://gitlab.com/gitlab-org/ux-research/-/issues/2465#note_1577201457). - **Continuously test user satisfaction with UXR and respond to their feedback** #### Developing a testing framework **Outcomes**: enabling efficient development and efficient contribution - https://gitlab.com/gitlab-org/gitlab/-/issues/422212+s - https://gitlab.com/gitlab-org/gitlab/-/issues/422245+s #### Monitoring quality / stability / speed / cost **Outcomes**: more trust production readiness of the chat - https://gitlab.com/groups/gitlab-org/-/epics/11234+ - Cost: https://gitlab.com/gitlab-org/gitlab/-/issues/415944+s / https://gitlab.com/gitlab-data/product-analytics/-/issues/1358+s - Error rate: https://gitlab.com/gitlab-org/gitlab/-/issues/421546+s tracks the error rate on individual 3rd party AI requests as well as error rate from the user perspective as responding to their request might require combining multiple 3rd party AI requests. @jprovaznik's assumes there are no errors beyond the 3rd party LLM service that would need to be considered (for instance between the backend and the client). The user level error rate calculation will interpret slow responses \>20 sec to be errors. The error rates will be tracked in Grafana. - Response time: <span dir="">\~\~</span> https://gitlab.com/gitlab-data/product-analytics/-/issues/1366+\\\\\\\\\\\\\\\~\\\\\\\\\\\\\\\~ Will be tracked in Prometheus on the worker level as part of https://gitlab.com/gitlab-org/gitlab/-/issues/421546+s by @jprovaznik. These measurements will include the time for all AI requests necessary to answer the user question but not the time to receive the user input from the frontend and sending the result back to the frontend. The difference should be marginal. - "Can't answer" rate: https://gitlab.com/gitlab-org/gitlab/-/issues/422202+s #### Other work items - [ ] https://gitlab.com/gitlab-org/gitlab/-/issues/420939+s (switch embeddings provider) - [ ] Mature the chat documentation and bring it to its own page. - [ ] Label the Chat as beta in the product and in the documentation. #### Integrate existing stand-alone AI features ??? - Should we integrate existing standalone AI features like _explain this code_ into the chat prior to the Beta release? - We should only do so if they reach the quality bar defined above. - What interdependencies are there with company announcements? # What the user can ask ### Questions related to current code Question | Tool test | Quality test ---|---|--- Can you explain the code? (Whole file or selected) Explain the selected code | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Can you explain function X? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Write me tests for function X | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | What is the complexity of the code | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | How would the code look like in Python? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | How would you refactor the code? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Can you fix the bug in my code | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Create an example of how to use function X | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Write documentation for the selected code | [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133770#note_1597887855) | Create a function to validate an e-mail address | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Create a tic tac toe game in Javascript | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | Create a function in Python to call the spotify API to get my playlists | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L222-246) | ### Questions related to the current issue or current epic Question | Tool test | Quality test ---|---|--- Please summarize the issue | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L126-144) | Summarize with bullet points | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L126-144) | Can you list all the labels? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L126-144) | How old is the epic? | [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133770/diffs#note_1597974580) | For which milestone is the issue? And how long until then? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L126-144) | Summarize the comments into bullet points? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/31d608f706ac72533502736939cdfe9c077c9975/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L142) | Questions on the actual content: "What should be the final solution for this issue?" | [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133770/diffs#note_1613052031) | ### Questions relating to the current MR Question | Tool test | Quality test ---|---|--- What is this MR about? | | Can you review this MR? | | Can you create a test for the new function in this MR? | | Can you simplify this code? | | ### Questions relating how to use GitLab Question | Tool test | Quality test ---|---|--- How do I change my password in GitLab? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L248-261) | How do I fork a project? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L248-261) | How do I clone a repository? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L248-261) | How do I create a template? | [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L248-261) | What is DevOps? What is DevSecOps? | [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133770/diffs#note_1597971516) | ### Follow-up questions #### Basic follow-up question that only require the last answer as context - Can you simplify your answer? - Context: there is at least one bot-answer in history. - Result: the last answer is simplified - Tool test: [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/b220eb42587282203362b59eafc3560b5578c9d6/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L167-218) - Quality test: - Can you translate your answer to German? - Context: there is at least one bot-answer in history. - Result: the last answer is translated - Tool test: - Quality test: - Can you explain your third point in different words? - Context: there is at least one bot-answer in history and the last answer contains a bulleted or numbered list (because that was the assignment of the previous question). - Result: the third point of the last answer is explained with different words. - Tool test: [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/3835ffd04808d5cc9bb1d52b93ce7b754092bfd7/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L382) - Quality test: #### Advanced follow-up questions that require the last answer + the context that was pulled for creating the last answer - Iterative in depth analysis of an issue - Context: there is at least one bot-answer in history and the last answer required the usage of a Tool to generate it. - Result: The answer contains information that was available in the result of the previously used Tool but not contained in any of the answers. - Example: - User is on: https://gitlab.com/gitlab-org/gitlab/-/issues/17800 - User asks: Can you identify the unique use cases the commenters have raised on this issue? - Bot: - uses `<issue tool to collet the issue context>` - answers `<with bulleted or numbered list of use cases>` - Tool test: [:white_check_mark:](https://gitlab.com/gitlab-org/gitlab/-/blob/3835ffd04808d5cc9bb1d52b93ce7b754092bfd7/ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb#L215) - Quality test: - User asks: Can you sort this list by the number of users that have requested the use case and include the number for each use case? Can you include a verbatim for the two most requested use cases that reflect the general opinion of commenters for these two use cases? - Bot: - uses `<history that includes the result of previously used tool but does not call the tool again>` - answers `<with sorted list that contains the number of users that asked for each use case and two verbatim>` - Tool test: [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134822/diffs#note_1613044754) - Quality test: - Can you expand on the last paragraph? - Context: there is at least one bot-answer in history and the last answer required the usage of a Tool to generate it. - Result: The answer contains information that was available in the result of the previously used Tool but not contained in any of the answers. - Tool test: [:x:](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134822/diffs#note_1613044367) - Quality test: - Can you detail the purpose of the code-owner class? - Context: there is at least one bot-answer in history and the last answer required the usage of a Tool to generate it. - Result: The answer contains information that was available in the result of the previously used Tool but not contained in any of the answers. - Tool test: - Quality test: #### Advanced follow-up questions that require multiple recent answers + multiple contexts that were pulled for creating the last answer (NOT PART OF BETA) - Comparison of two issues - Example - History: continued from above big example "Iterative in depth analysis of an issue " - User switched to: https://gitlab.com/gitlab-org/gitlab/-/issues/12104 - User asks: How are the use cases raised in this issue different or the same? - Bot: - uses `<issue tool to collet the issue context for 12104>` - uses `<history that includes the result of previously used tool to get context for 17800 but does not call the tool again>` - answers `<with two list: one with similarities and one with differences>` - Tool test: - Quality test: #### Questions that the user did not formulate clear enough should result in a question from the AI to clarify the question - User: `How do I create a template?` - Although this question seems to be reasonable, it's not clear what kind of template is meant. I think the only solution for such questions is to have a means for the AI to ask a clarifying question, for example the following - Bot: `The term template is used widely in the context. Could you be more specific which kind of template you would like to create?` - Tool test: - Quality test: ## What will not be supported * We will not support other languages than English. That does not mean it won't work, but it will not have been tested at all. We made this decision to prevent a significant increase in complexity. * We will not support comparing and relating different contexts to another. E.g. what is the difference between this issue and issue xyz. This is GA scope. * We will not support global contexts for repos / issues / epics / etc. This may be covered before GA though. ## Marketing * https://gitlab.com/gitlab-com/marketing/brand-product-marketing/product-marketing/-/issues/7274+s * Video intro for blog post: https://youtu.be/l6vsd1HMaYA * Blog post: https://about.gitlab.com/blog/2023/11/09/gitlab-duo-chat-beta/ / https://docs.google.com/document/d/1hZgDwqKbcRvED-ws_VlmeLzpkbN5vU6rRleejm1OLXc/edit * Press release: https://about.gitlab.com/press/releases/2023-11-09-gitlab-continues-ai-momentum-with-enhanced-ai-powered-devsecops-workflows.html <details><summary>Plain text from the blog post to help AI generate a good release post</summary> GitLab Duo Chat, the newest feature in the GitLab Duo suite of AI-assisted capabilities, helps teams write and understand code faster, get up to speed on the status of projects, and quickly learn GitLab. Chat will be available in Beta starting November 16. Chat also will be included in our Web IDE and GitLab Workflow extension for VS Code as an experimental release. Get code explanations, generate tests, and create code right where your development work happens; no need for context switching. We launched the GitLab Duo suite earlier this year to bolster software development teams' workflow, helping you deliver more secure software at an unprecedented pace and create more value for your customers. GitLab is the only platform that brings AI-powered planning tools, code creation, security scanning, and vulnerability remediation all into a single developer-friendly experience. Chat will serve as the foundational technology driving our AI-powered GitLab Duo features, such as Vulnerability Summary and Root Cause Analysis. This will enable you to pose in-context follow-up questions for a more thorough investigation In our Global DevSecOps Report: The State of AI in Software Development, 83% of respondents said that implementing AI in their software development processes is essential to avoid falling behind, and they ranked chatbots among their top three AI use cases. Chat, which will be released for Ultimate tier customers as part of GitLab 16.6, is the perfect feature to help you maintain your competitive advantage. Here's why. Designed to support everyone involved across the software development process From coding assistance to productivity tips, Chat provides real-time support for technical and non-technical users across the entire software development lifecycle. Inspiration on demand: Need help determining the next best step in your workflow? Chat is ever-ready to ignite your ideation process. Elevate productivity: Chat shoulders the burden of routine tasks so you can channel your energy into delivering value to your customers. Guidance at every step: Whether you're a GitLab expert or a newcomer, Chat is your go-to coach, helping you become an expert in any process or feature. Sometimes, there isn't enough time in the day, and even though you want to focus on important tasks, there are many little things you need to do. Code assistance: Chat can assist in decoding the mysteries of unfamiliar code. It can explain, propose tests, or simplify the code. You can also use the Chat to write code from scratch interactively. Issue and epic management: Summarize an issue in seconds, turn comments into an issue description, or distill specific information from large epics easily. You can ask any question about the content of an epic or an issue. Plan work with issues and epics faster with the help of Chat. Onboarding and learning made simple: Whether you are onboarding to GitLab or you are already an expert learning how to use GitLab is streamlined with Chat. How your data stays your data GitLab Duo Chat does not use your proprietary code or inputs to Chat as training data. This privacy-first approach includes both the prompt and the output of Chat. GitLab Duo uses the right large language models (LLMs) for each use case. For instance, Anthropic Claude-2 and Vertex AI Codey with text embedding-gecko LLMs power Chat. Our publicly available documentation describes all AI models GitLab Duo uses and how it uses your data. The road ahead for GitLab Duo As we continue to innovate and improve GitLab Duo, we're excited to share that our Code Suggestions capability will transition from Beta to general availability later this year. We look forward to seeing the transformative impact it will have on your software development efforts. Learn more about GitLab Duo Chat in our documentation and share your feedback and ideas. </details> ## Related * [Research work prioritization for the GitLab Chat in Q2 FY24](https://gitlab.com/gitlab-org/ux-research/-/issues/2501#note_1421246798 "Research Prioritization for Fulfillment, ModelOps, and Analytics - Q2FY24") * https://gitlab.com/gitlab-org/ux-research/-/issues/2604#note_1515310932 * https://gitlab.com/gitlab-org/gitlab/-/issues/423325+s * https://gitlab.com/gitlab-org/gitlab/-/issues/438176+s ## Literature (some random links) * Research result: [Questions users want to ask](https://gitlab.dovetailapp.com/projects/6wL9UBVQCbOpaC6WLu22su/v/ukXfzRL2lbYwiKe5Bbdyu/present/1WktHKngt2oO4NpHzBEf9I) * [What parts of development workflow make sense for AI tools](https://stackoverflow.co/labs/developer-sentiment-ai-ml/#h2-what-parts-of-development-workflow-make-sense-for-ai-tools) * https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm * https://newrelic.com/platform/new-relic-grok * https://twitter.com/amanrsanger/status/1660729100137553920 * Test framework for code suggestions: https://gitlab.com/gitlab-org/gitlab/-/issues/415381+