Live traces could be missing when failed to create trace artifact records
Summary
Currently, all live trace files have been moving from the original location(#{YYYY_MM}/#{project.id}/#{job.id}
), but if the creation of trace artifacts was failed by exceptions (DB error, AR validation failure, etc), the original files will be gone from the previous permanent path and remains in tmp/cache
. This is because we don't catch the exceptions nor restore the original file from tmp/cache
.
The impact
Live trace files could be gone
What we should do at first
Disable CreateTraceArtifactWorker
queues from production (gitlab.com) until the hotpatch is applied. This should not give any impacts to users.
The right solution
We always work on temporary files, like the following way
def execute(job)
return if job.job_artifacts_trace
job.trace.read do |stream|
return unless stream.file?
build_log = Tempfile.new('build.log')
build_log.close
FileUtils.cp(stream.path, build_log.path)
job.create_job_artifacts_trace!(
project: job.project,
file_type: :trace,
file: UploadedFile.new(
build_log.path,
'build.log',
'application/octet-stream'
)
)
FileUtils.rm(stream.path)
ensure
build_log&.close
end
/cc @ayufan @ahanselka
Edited by Shinya Maeda