Skip to content

Add evaluation for documentation questions with context

Gosia Ksionek requested to merge mk-add-documentation-to-evaluations into main

What does this merge request do and why?

I have hound this dataset - we need to check if it is used/fresh: https://console.cloud.google.com/bigquery?ws=!1m5!1m4!4m3!1sdev-ai-research-0e2f8974!2sduo_chat!3sdocumentation_v3&project=ai-enablement-dev-69497ba7

and I imported it to langsmith, also created custom prompt for evaluation (I used claude generate prompt function, it’s brilliant).

result: https://smith.langchain.com/o/477de7ad-583e-47b6-a1c4-c4a0300e7aca/datasets/88d6470e-e974-45d6-b63a-5781391f6653/compare?selectedSessions=1f871741-138b-4113-9a38-21f1f585e2d1&baseline=undefined

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Merge request reports