If a live trace has already been archived, remove it

Problem

When a small race condition happens, both a live trace and an archived trace exist on a single Ci::Build record. Usually, a live trace will be removed after it's archived.

In this case, ::Gitlab::Ci::Trace::AlreadyArchivedError exception is raised when the system tries to archive the build. We currently silently ignore those errors, however, ideally, we should remove those unnecessary live traces.

For example, there are some build with this condition on dev.gitlab.org

irb(main):009:0> Ci::Build.finished.with_live_trace.with_archived_trace.order(:created_at).map { |b| puts "#{b.id}: #{b.created_at}" }
2431274: 2018-05-14 11:56:27 UTC
2431275: 2018-05-14 11:56:27 UTC
2431276: 2018-05-14 11:56:27 UTC
2431307: 2018-05-14 12:13:40 UTC
2431308: 2018-05-14 12:13:40 UTC
2431309: 2018-05-14 12:13:40 UTC
2431310: 2018-05-14 12:13:40 UTC
2431311: 2018-05-14 12:13:40 UTC
2431312: 2018-05-14 12:13:40 UTC
2431313: 2018-05-14 12:13:40 UTC
2431314: 2018-05-14 12:13:40 UTC
2433845: 2018-05-15 02:19:33 UTC
2435250: 2018-05-15 11:40:53 UTC
2435777: 2018-05-15 15:43:41 UTC
2436827: 2018-05-16 14:08:38 UTC
2436828: 2018-05-16 14:08:38 UTC
2436829: 2018-05-16 14:08:39 UTC
2436830: 2018-05-16 14:08:39 UTC
2437034: 2018-05-16 15:01:21 UTC
2437987: 2018-05-17 07:54:41 UTC
2438491: 2018-05-17 21:04:08 UTC
2438498: 2018-05-17 21:04:09 UTC
2438568: 2018-05-17 21:04:52 UTC
2438586: 2018-05-17 21:04:53 UTC
2438970: 2018-05-17 21:09:11 UTC
2439731: 2018-05-18 07:00:07 UTC
2440296: 2018-05-18 09:59:50 UTC
2440982: 2018-05-18 11:27:18 UTC
2441068: 2018-05-18 11:28:08 UTC
2446626: 2018-05-21 12:29:29 UTC
2447983: 2018-05-21 16:05:21 UTC
2450086: 2018-05-22 05:11:54 UTC
2450959: 2018-05-22 10:01:02 UTC
2452597: 2018-05-23 07:19:53 UTC
2452598: 2018-05-23 07:22:05 UTC
2453384: 2018-05-23 09:41:52 UTC
2454424: 2018-05-23 12:50:42 UTC
2454609: 2018-05-23 12:56:46 UTC
2455543: 2018-05-23 18:49:17 UTC
2458103: 2018-05-24 14:28:13 UTC
2458157: 2018-05-24 14:28:15 UTC
2458196: 2018-05-24 14:28:25 UTC
2458230: 2018-05-24 14:28:26 UTC
2458294: 2018-05-24 14:43:27 UTC
2458522: 2018-05-24 14:43:50 UTC
2458588: 2018-05-24 14:49:06 UTC
2458595: 2018-05-24 14:49:06 UTC
2458633: 2018-05-24 14:49:11 UTC
2458719: 2018-05-24 14:52:45 UTC
2460205: 2018-05-25 14:50:25 UTC
2461138: 2018-05-25 17:09:21 UTC
2461139: 2018-05-25 17:09:21 UTC
2461140: 2018-05-25 17:09:21 UTC
2461494: 2018-05-25 20:04:38 UTC
2461495: 2018-05-25 20:04:38 UTC
2461496: 2018-05-25 20:04:38 UTC
2461497: 2018-05-25 20:04:38 UTC
2461498: 2018-05-25 20:04:38 UTC
2461499: 2018-05-25 20:04:38 UTC
2461500: 2018-05-25 20:04:38 UTC
2461501: 2018-05-25 20:04:38 UTC
2461502: 2018-05-25 20:04:38 UTC
2464266: 2018-05-28 11:05:26 UTC
2464646: 2018-05-28 11:08:23 UTC
3087662: 2018-09-13 15:27:42 UTC
=> [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]
irb(main):010:0> 

We can remove live trace (not archived trace) from those builds

Impact

There are no immidiate threats from those builds.

It's a bit bothering that makes the cron worker ArchiveTracesCronWorker iterate on those builds every hour.

Solution

TBD