Fix missing pipeline e-mails when job logs moved to object storage
As discussed in #195430 (closed), it's possible that a race condition occurs when a build finishes and a failed pipeline email goes out. This is what was happening before:
-
Build finishes and causes a pipeline failure. The pipeline failure transition causes the pipeline to run
PipelineNotificationWorker
, which schedules anActiveJob
to e-mail a user with a failed pipeline ID. -
BuildFinishedWorker
schedulesArchiveTraceWorker
to move the job log, stored locally on a filesystem, to object storage. -
The
ActiveJob
runs and loads the pipeline and failed build logs. Some builds have been moved to object storage, but theActiveJob
has a stale record and attempts to load the file from the filesystem. -
The file doesn't exist, an
Errno::ENOENT
exception is raised.
To fix this, we now attempt a refresh the job from the database if we encounter this exception.