Optimize Releases::ManageEvidenceWorker

Summary

Releases::ManageEvidenceWorker runs every hour to collect release evidence for upcoming releases. However, sometimes it fails with a database statement timeout while trying to fetch the relevant CI pipeline.

When this exception is raised, the job fails completely and any releases that have yet to be processed are skipped, even though they may not require any slow/expensive queries.

Relevant logs and/or screenshots

  • https://new-sentry.gitlab.net/organizations/gitlab/issues/285162/ [outdated]
  • Logs
  • Query: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/33301/commands/102749
  • Context from MR: !171011 (comment 2198232643)

Possible fixes

  • If possible, optimise Releases::EvidencePipelineFinder to stop queries reaching the statement timeout.
  • If Releases::EvidencePipelineFinder fails, or another failure is encountered creating evidence for a release, skip and continue with the rest of the release records instead of aborting the job.
Edited Nov 07, 2024 by Andrew Jung
Assignee Loading
Time tracking Loading