Investigate running of SWE bench for Duo Workflow

marked this issue as related to #29 (closed)

@achueshev - created this issue based on your suggestion from the last sync up call. Sounds like a good idea. Do you plan to investigate this option further or should I take this one (both is fine by me)? I think it should be feasible to either import projects from github or re-create only needed resources - issues/comments, similar to draft in gitlab-org/duo-workflow/testing/duo-workflow-tests!1 (closed).

/cc @pwietchner @bastirehm

@pwietchner, @jprovaznik, @achueshev:

Wdyt about just focusing on SWE bench lite for now? From what I understand the full SWE bench will likely be time consuming and costly to run. It also can contain issues that contain references to other issues / commit-sha's which might be difficult to replicate.

The criteria that have been used to select the SWE bench lite subset of problems should make them ideally suited to just be imported into GitLab.

@bastirehm sounds good to me

Do you plan to investigate this option further or should I take this one (both is fine by me)?

@achueshev as I had some spare time I started on a POC MR which adds support for SWE to https://gitlab.com/gitlab-org/duo-workflow/testing/duo-workflow-tests

Quick update:

We had a sync meeting today with @jprovaznik to discuss how we can collect SWE metrics. Please find the outcome and key points below:

We will start with the SWE-bench Lite dataset, selecting the dev split that contains 28 examples: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite
@jprovaznik has already moved the selected SWE dataset to LangSmith
We need to update the evaluation script in https://gitlab.com/gitlab-org/duo-workflow/testing/duo-workflow-tests to:
1. Read the dataset from LangSmith
2. Run Duo Workflow with the prompt containing the problem statement Note: Before running Duo Workflow, we need to clone related projects and checkout the base_commit.
The evaluation script should return patches generated in the following format: https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240415_eval_bug/check-harness.jsonl

Once we obtain Duo Workflow results, we need to run the SWE harness script with the following commands:

git clone git@github.com:princeton-nlp/SWE-bench.git
cd SWE-bench
<please create a virtual environment to avoid breaking your system Python, with `virtualenv`>
pip install -e .
python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Lite \
  --split=dev \
  --predictions_path <path_to_DW_results> \
  --max_workers <num_workers>

cc @oregand @pwietchner

@jprovaznik , let me know if you need help running the SWE script.

@achueshev I've updated test run to support also SWE and return results in expected format. It's possible to run them just by setting SWE dataset, e.g.:

docker run -e DATASET=duo_workflow.swe-bench-lite.1 -e WORKFLOW_CONCURRENCY=8 -e USE_MEMSAVER=1 -e LANGCHAIN_API_KEY=`cat ../token-langsmith` -e ANTHROPIC_API_KEY=`cat ~/Downloads/anthropic-eval-key.txt` -e GITLAB_URL="https://gitlab.com" -e WORKFLOW_RECORD_GITLAB_REQUESTS=1 --mount type=bind,src=/home/honzap/gitlab-development-kit/duo-workflow-tests/,target=/opt/tests duo-workflow-test

let me know if you need help running the SWE script.

Unfortunately I'm hitting an error if I try to evaluate these results now. When running:

python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path ../duo-workflow-tests/run-2024-10-03-09-08-18/results.jsonl --max_workers 1 --run_id "test1" --split=dev

It fails to build docker image, content of image build log:

2024-10-03 12:50:21,223 - INFO - ERROR: No matching distribution found for types-pkg_resources
2024-10-03 12:50:22,699 - INFO - ---> Removed intermediate container cfa46bb9b55e
2024-10-03 12:50:22,699 - ERROR - Error: The command '/bin/sh -c /bin/bash -c "source ~/.bashrc && /root/setup_env.sh"' returned a non-zero code: 1
2024-10-03 12:50:22,699 - ERROR - docker.errors.BuildError during sweb.env.x86_64.29ea89393f6403e072c9f5:latest: The command '/bin/sh -c /bin/bash -c "source ~/.bashrc && /root/setup_env.sh"' returned a non-zero code: 1

Which is a known issue - https://github.com/princeton-nlp/SWE-bench/issues/199

Didn't investigate this further - let me know please if you hit this one.