Optimize Releases::ManageEvidenceWorker
Summary
Releases::ManageEvidenceWorker runs every hour to collect release evidence for upcoming releases. However, sometimes it fails with a database statement timeout while trying to fetch the relevant CI pipeline.
When this exception is raised, the job fails completely and any releases that have yet to be processed are skipped, even though they may not require any slow/expensive queries.
Relevant logs and/or screenshots
- https://new-sentry.gitlab.net/organizations/gitlab/issues/285162/ [outdated]
- Logs
- Query: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/33301/commands/102749
- Context from MR: !171011 (comment 2198232643)
Possible fixes
- If possible, optimise
Releases::EvidencePipelineFinderto stop queries reaching the statement timeout. - If
Releases::EvidencePipelineFinderfails, or another failure is encountered creating evidence for a release, skip and continue with the rest of the release records instead of aborting the job.
Edited by Andrew Jung