Self-hosted models: Evaluate from local documentation index
Problem to solve
We understand the local document search performs worse than embeddings or VertexAI search, but we need to know how much worse to define expectation and improvements
Solution
For each supported Chat model, run the GitLab Documentation eval dataset using local search as the backend instead of VertexAI.
Document the results of each supported model for Chat against the documentation eval dataset within this issue.
Edited by Susie Bitters