Null Value analysis and variance check of Chat API and Chat UI
🔦
Objective The Primary Objecive it to investigate the null value analysis and understand if Chat UI and API are consistent through the process as well.
#⃣
Primary Metric for Success The primary metric for success for this experiment is having more than 0.50 similarity score with Chat having an answer
📚
Dataset for Diagnostic Testing/Experimentation Here is a small subset of data for experimentation: https://docs.google.com/spreadsheets/d/1IZ8DU3TUlrPebfAm1LBURyzpQ47s46VNyiPoaQccfH4/edit#gid=0
We have created 3 videos to walkthrough the idea behind this change and an example how to incorporate it to Duo Chat development
- How to run end to end experiments as an example of trimmed prompts (https://www.youtube.com/watch?v=H2oykA5THac)
- Curious on how the datasets are build and run the experiments (https://www.youtube.com/watch?v=swN2EtAzdWA)
- A generic walkthrough of end to end pipeline to get familiar with accessing Prompt Library( https://youtu.be/U2CW95yylMs)
🔍
Metrics - Control Metric Score: Similarity Score Average 0.52
- Experiment Metric Score: TBD post Experiment
- Variance:
📶 : TBD Post Experiment
✍🏼
Experiment Details Recommendation: consider investigating if there is a variance in Chat API and Chat UI and if Chat UI is still responding with these questions?
Edited by Mon Ray