As we are adding more AI features and prompts rapidly we are expecting that the amount of evaluation needs will grow exponentially. For development of those and constant iteration we need to have an evaluation setup for both individual prompts and complete features that is easy to use, owned by the specific teams and is scalable to the growing amount of evaluations so we can safely iterate and upgrade prompts and models. ### Setup today * Daily production runs of the CEF (Central Evaluation Framework) evaluating Duo Chat reporting into 1 central dashboard * Setup to run evaluation datasets locally * Datasets managed by Evaluation team * API / Model based code suggestions evaluation ### Things to solve * No  iteration + evaluation setup for single prompts possible * Prompts amount will increase exponentially in Chat Agents and Workflow Agents * We only can make changes in the main code base to a prompt and then evaluate chat as a whole and not a single subpart in the first iteration step * We tried with a colab notebook but very limited and manual coding is needed * Iteration of Chat changes are very hard * Setup of local evaluation takes up to 22 steps and takes a while with very limited amount of steerability * Discovery of correct datasets is hard and datasets owned by evaluation framework * Full dataset runs take a while and can only manually be triggered by evaluation team * Missing other evaluation criteria like conciseness, coherence, latency, tokens, hal etc.  * No Side by side evaluation possible which is crucial for prompt changes and especially model changes/upgrades/comparison with actual prompts which will be even more important with self hosted models * We have no traceability of bad results , which leads to multi hour (or even days) to reproduce and find the source of the problem * With the planned increase on Chat Agents and its use cases the amount of evaluations will increase by a lot which won’t scale with the current evaluation workflow. * Also datasets owned by the respective teams should help iterate those and distribute workload * Code suggestions evaluation as a whole solution with all aspects is needed and not only on an API/model level * Code suggestions Evaluation doesn't have evaluations or a setup regarding evals for a full codebase to test context relevant evaluation * Scaling and Speed of getting Evaluation tooling ### Suggested solution for Chat + other AI features * Teams can setup and edit prompts, datasets, experiments and evaluation runs on their own * Atom and Molecule testing (both single prompts and full features) * 1 step runs for local evaluation of features * Constant evaluation which includes also Tracing ### Evaluation Results | Issue | Result | Comments | |-------|-----------|----------| | Setup and editing of datasets | TBA | Teams can manage their own datasets and make quick modifications without central control. For example, setting up a dataset to evaluate chat prompts can be done in a few clicks. See https://smith.langchain.com/o/477de7ad-583e-47b6-a1c4-c4a0300e7aca/datasets/2a6710c7-6b8c-4dd2-8d78-c38e58078bf3?tab=2 as example of a Chat documentation dataset, and [code generation dataset working on langsmith](https://smith.langchain.com/o/477de7ad-583e-47b6-a1c4-c4a0300e7aca/datasets/290eb1bf-b782-4883-8973-2ba0b2c778cb/compare?selectedSessions=d3a1f8a5-2103-4ae0-bfc4-7c85b585449c&baseline=undefined). | | Iteration + evaluation setup for single prompts | TBA | LangSmith allows for easy and rapid iteration on single prompts. Teams can test individual prompts against small datasets and quickly make adjustments. See https://smith.langchain.com/prompts/duo_chat_issue_tool?organizationId=477de7ad-583e-47b6-a1c4-c4a0300e7aca as an example, also see https://gitlab.slack.com/archives/C06LWENL58F/p1719242481567359 and https://gitlab.enterprise.slack.com/files/U06K0PJDXR6/F079GQV4GN7/screen_recording_2024-06-24_at_10.17.50___am.mov | | Easy local runs of evaluation against feature | TBA | LangSmith supports one-step local evaluation runs, significantly reducing the complexity and time required for local testing. See https://gitlab.com/gitlab-org/ai-powered/eli5/-/tree/main/evaluation_scripts/chat?ref_type=heads and https://gitlab.com/gitlab-org/ai-powered/eli5/-/merge_requests/8 as examples | | Adding other evaluation criteria like conciseness, coherence, latency, tokens, hal etc. | TBA | LangSmith offers a wide range of built-in evaluation criteria and allows for the addition of custom metrics. See https://gitlab.com/gitlab-org/ai-powered/eli5/-/blob/main/doc/img/new_experiment.png?ref_type=heads as an example | | Side by side evaluation | TBA | TBA | | Tracing of bad results | TBA | LangSmith includes detailed tracing features that help in identifying and analyzing poor performance or errors in prompt responses. See https://gitlab.com/gitlab-org/ai-powered/eli5/-/issues/1#note_1962238859 as an example | | Time to setup a dataset and full evaluation pipeline | TBA | TBA | | Custom evaluators | TBA | LangSmith allows for the easy creation and integration of custom evaluators. See https://gitlab.com/gitlab-org/ai-powered/eli5/-/tree/main/doc/evaluators?ref_type=heads#langchain-evaluators as an example | #### Use an existing LLM Evaluation / Tracing / Prompt registry solution * Used features * Web UI + Usage of SDK to integrate into CLI GDK for full local runs * [Tracing of LLM calls/chains](https://docs.smith.langchain.com/concepts/tracing) for local runs and test instances * [Dataset collection + management](https://docs.smith.langchain.com/how_to_guides/datasets/manage_datasets_in_application) * Slicing of datasets * [Prompt management](https://docs.smith.langchain.com/how_to_guides/prompts/create_a_prompt) + [iteration workbench](https://docs.smith.langchain.com/how_to_guides/evaluation/run_evaluation_from_prompt_playground) * [Testing + Evaluation setup ](https://docs.smith.langchain.com/concepts/evaluation) * to run the changed prompts or whole applications (chat, other AI features) * with different models or model settings * A lot of fifferent existing evaluation methods and custom methods * Solution suggested after evaluation for workflow work : [LangSmith](https://docs.smith.langchain.com/pricing) (Plus is 39 USD/user/month, for Enterprise we need to contact , \~40 users?) * Evaluation happened by AI Framework team, [short intro video](https://gitlab.slack.com/archives/C06LWENL58F/p1715265894527579) * Existing tracing for Chat already - https://docs.gitlab.com/ee/development/ai_features/duo_chat.html#tracing-with-langsmith  * Would extend our evaluation capabilities and the actual workflow by a huge jump instantly * [Off the shelf evaluators](https://docs.smith.langchain.com/reference/sdk_reference/langchain_evaluators) * Custom evaluators * Evaluation Cookbook around a lot of different scenarios https://github.com/langchain-ai/langsmith-cookbook/blob/main/README.md#testing--evaluation * Allows us to test/evaluate atoms (single prompts) and molecules (full application parts) * Scales easily to all AI efforts as we add more capabilities to chat, app and the workflow solution * Lets us much easier evaluate different models on the single prompts which would be important for custom models evaluation * Datasets, Evaluations and executions can be owned by each respective team * Immediate insights when something doesn’t work through tracing * Import existing test datasets #### Development workflow for Chat / AI Features / Duo Workflow Agents * Each prompt is saved in the Langsmith model registry  * We implement a prompt fetching mechanism in all systems (Rails monolith, AI gateway, etc.) that has a pure offline version in the background (if SM can’t update online) but can update later at any time from the online source (along the already existing Langsmith registry functionality) * Might need to have an step in between for exporting and updating the offline version to each codebase * Local development setup is connected to tracing of the tool to have full insights into the execution , like we have today Langsmith integration for Chat already  * Groups themselves are responsible for their respective evaluation datasets and can manage them easily (Development, Test Engineers, Product Managers) * Central guidance on how datasets and experiments should be setup **Development Workflow** 1. IC iterate in Tools workbench the specific single prompt (e.g. 0 shot agent in Chat, prompts for each chat agent, prompts for other AI features, workflow agents/tools prompts) and constantly evaluate with a small dataset and specifically selected evaluators those prompt changes  2. IC do a side-by-side run/comparison with the current prompt (or setting or model) and the updated prompt/model/settings.  3. IC can update with a single command the prompt in the actual application locally and then start with a CLI an evaluation test run that allows them to select the evaluation task, the specific dataset (for example different sizes), full evaluation with selected evaluators for that tasks, latency, token usage, etc and especially tracings from each single evaluation land in Langsmith. When local results are fine committed and submitted as MR 1 line CLI (`gdk evaluate chat`) lets the user choose from one of the experiment setups (datasets+evaluators) and execute an evaluation run with local instance Trace of one of the answers in the test data set that failed on the evaluation: Same evaluation as locally could be triggered also in CI job **Daily testing of the current solution** We deploy to a specific test instance on a reference architecture, we setup there also tracing to Langsmith and run the largest evaluation dataset for each application part (Chat, AI features, etc. ) against this test instance. Should give us the same results as production but allows us to have immediately full tracings on each single test run which should make debugging and improvements much faster. ### Suggested solution for code suggestions \ [Extend the competitive analysis tool](https://gitlab.slack.com/archives/C068AMYRUUF/p1715156423738379) by the test team with more evaluation datasets (especially regarding context fetching) as this setup gives us way more data of the application as a whole compared to pure model or API evaluation. We found for example [quickly a bug based](https://gitlab.com/gitlab-org/gitlab/-/issues/461081#note_1903323918 "Review potential performance/ code quality issues in Code Suggestion analysed by the comparator tool") on the wide array of data we got in code creation. We need to produce quick evaluation results between Duo and Co-pilot in each iteration of; underlying models, updated test data and improvements to our AI architecture.