### Problem Currently, our only means of measuring the chat's answer quality is via manual testing and regular [Chat Bashes](https://gitlab.com/gitlab-org/ux-research/-/issues/2513) with users. This is slow, never covers the breadth of the chat's capabilities, and is not repeatable. So, developers are rather blind when attempting to improve the chat quality. (I use the word quality here as I don’t have a better word that would cover accuracy and helpfulness in one.) They may be aware of an issue and have a good plan to fix the issue, but are unaware whether they break some other capability of the chat. ### Goal Goal is to have a test framework - that lets team members and contributors make changes to prompts and other things and see how these changes affect the answer quality. - that let's us understand the status quo of the chat and informs prompt engineering to improve the chat. - that let’s us monitor the chat, so that we know when things worsen. Ideally, the test framework is reflective of the users' needs in terms 1. the questions they are likely to ask 1. the way they might formulate these questions 1. the data that they would like the questions to be applied to 1. the desired content of the answer and the way the answer is formulated ### Proposal #### Assumptions Through our regular [Chat Bashes](https://gitlab.com/gitlab-org/ux-research/-/issues/2513) we know that we are currently [far from satisfying the users](https://app.periscopedata.com/app/gitlab/1137231/Ai-Features?widget=17243436&udv=1959424). From our own manual tests we know that the LLM itself (when given the right data) does a much better job at creating a good answer than the chat does ([see an example](https://gitlab.com/groups/gitlab-org/-/epics/11567#note_1582569249)). The reason is that the chat involves many layers to identify the right data and multiple interactions with the LLM to draft the final answer. There are many opportunities along the way to fail. So, it happens that the chat summarizes or explains the wrong content, or simply chooses the wrong tool. Therefore, we are going to make an assumption that the LLM can produce "known good answers" (or ground truth) for us so that we can compare the chat answer to these. So, this assumption will stand in for the above mentioned users's need to have the answer contain the desired content and be formulated appropriately (see bullet 4 above). ##### IMPORTANT: Once the chat itself does a similarly good job as the LLM on its own, this assumption will no longer be helpful when attempting to further improve user satisfaction. We have to reconsider when we get to that point. It may well be that the answers are good enough and need no further improvement. #### Observations Through tinkering we found that the chat can fail in many ways such as: * Small changes in the capitalization of individual words in the user question can make a difference between success and failure; although the LLM itself is very robust to these changes. * Small changes in way the questions is formulated can also make a big difference in the result; again the LLM itself is rather robust to the formulation. * The chat sometimes fails to create the correct resource id and thus fails to collect the (right) data. * Many users are none-native English speakers and thus formulate the questions poorly which may further increase the probability to fail; again the LLM itself is rather robust to such variations. #### Proposal 1. Collect questions relevant to the users and collect typical data that they want the questions to be applied to. - As we cannot collected this from chat usage itself for privacy reasons, we will use the [Chat Bashes](https://gitlab.com/gitlab-org/ux-research/-/issues/2513) to inspire this collection. - This data is collected in the [Chat test cases](https://docs.google.com/spreadsheets/d/1jm059l3T9XL2aj5VgBsqVgcumbiqYDZ0c-Utyrg0u-o/edit#gid=0) spreadsheet. 2. Take these user questions and create permutation of the way the questions are formulated and spelled. - Use an LLM to generate these permutations. - Then take those permutations and create further permutations instructing the LLM to write the question as people with different language background would write them in English. - Then take all these permutations and create further permutations that introduce different capitalization, punctuation and spelling mistakes. 3. Take the context data (issues, epics, code) that users have applied the questions to in the Chat Bashes to inspire a larger collection of context data to combine the questions with. 4. For all combinations of such context data and permutated questions generate known good answers. - Use an LLM (or multiple LLMs) to generate a set of known good answers and their embeddings. - [Here](https://docs.google.com/spreadsheets/d/1jm059l3T9XL2aj5VgBsqVgcumbiqYDZ0c-Utyrg0u-o/edit#gid=0) is an example how that could look like based on a hand-crafted set of three questions at the top on the first tab "Questions relating to issues and epics". 5. Run the questions and the context data through the chat (we need to build a [Chat API](https://gitlab.com/gitlab-org/gitlab/-/issues/426619)) and test the chat on three levels: 1. Check if the chat picks the right tool to answer the question 2. Check if the chat generates the correct resource ID to collect the right context data 3. Compare the answers and their embeddings with the embeddings of the known good answers. - Check if the chat's responses come close to these answers using matching scores / matching algorithms ~~e.g. by comparing cosine similarity or cross similarity~~ - Potentially involve more [elaborate ways of assessing the answer](https://levelup.gitconnected.com/evaluation-driven-development-the-swiss-army-knife-for-rag-pipelines-dba24218d47e) as an advanced iteration. 6. Go through the results (i.e. data science work) to identify patterns how the chat fails and improve the prompts to get better. 7. Run a subset of these potentially thousands of tests before each merge to see if the change leads to a degradation. 8. Also run this subset of questions in a regular monitoring job. #### How to slice the solution Initially, we had intended to start with issue related question and check the chat on all three levels mentioned above (see bullet 5) and only then add other types of questions. Having understood that a common failure is picking up the wrong tool, it seems more appropriate to start with questions of all types but iterate over the three levels mentioned above. ### Related links and issues * https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/prompt-library/-/issues/147+