Add retry to rake task
What does this MR do and why?
While running duo_chat:completions rake task, we may run into not-code-related errors, such as network errors, timeout errors. It is common. With retry, we can run the whole rake task without needing to restart after every problem.
Logging allows to understand if the error is intermittent or not.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Run rake task:
AIEF_DATASET=/Users/gosiaksionek/datasets/duo_chat/v1/jsonl/queries \
AIEF_OUTPUT=/Users/gosiaksionek/duo_chat_completions GITLAB_SIMULATE_SAAS=true AI_GATEWAY_URL=http://0.0.0.0:5052 AIEF_GITLAB_BASE_URL=http://127.0.0.1:3000 bundle exec rake "gitlab:duo_chat:completions[duochateval, 1]"Please follow this guides to run rake task: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/prompt-library/-/blob/main/doc/how-to/run_duo_chat_eval.md#seeding-the-local-gdk-instance-with-issue-and-epics-data