Skip to content

Add assessment to the SWE flow

What does this merge request do and why?

This MR adds the assessment section to the swe flow YAML eval config to successfully run swebenchmark.

BLOCKED by !1759.

How to set up and validate locally

Check the docs updated with this MR.

Output example - https://smith.langchain.com/o/477de7ad-583e-47b6-a1c4-c4a0300e7aca/datasets/6cd898d8-3b3c-49d4-bfd5-944f83bea1f2/compare?selectedSessions=2e57f507-0628-48a8-9c9e-93ee6abf315f&baseline=undefined

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Alexander Chueshev

Merge request reports

Loading