Live traces could be missing when failed to create trace artifact records

Summary

Currently, all live trace files have been moving from the original location(#{YYYY_MM}/#{project.id}/#{job.id}), but if the creation of trace artifacts was failed by exceptions (DB error, AR validation failure, etc), the original files will be gone from the previous permanent path and remains in tmp/cache. This is because we don't catch the exceptions nor restore the original file from tmp/cache.

The impact

Live trace files could be gone

### What we should do at first

Disable CreateTraceArtifactWorker queues from production (gitlab.com) until the hotpatch is applied. This should not give any impacts to users.

The right solution

We always work on temporary files, like the following way

def execute(job)
  return if job.job_artifacts_trace

  job.trace.read do |stream|
    return unless stream.file?

    build_log = Tempfile.new('build.log')
    build_log.close
    FileUtils.cp(stream.path, build_log.path)

    job.create_job_artifacts_trace!(
      project: job.project,
      file_type: :trace,
      file: UploadedFile.new(
        build_log.path,
        'build.log',
        'application/octet-stream'
      )
    )

  FileUtils.rm(stream.path)
ensure
  build_log&.close
end

/cc @ayufan @ahanselka

• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17313