Ci::ArchiveTracesCronWorker keeps failing false positively

Summary

Ci::ArchiveTracesCronWorker is a cron worker to archive stale live traces.

Today, I found that job_trace_archive_failed_total metric on Prometheus keeps rising on dev.gitlab.org.

https://prometheus-app-01.gitlab.net/graph?g0.range_input=1w&g0.expr=job_trace_archive_failed_total&g0.tab=0&g1.range_input=1h&g1.expr=pages_deployments_failed_total&g1.tab=0

Here is the corresponding log on Kibana

https://log.gitlab.net/app/kibana#/discover?_g=h@91bd11d&_a=h@fae6573

The problem is that the worker tries to archive job traces which are just waiting for BuildFinishedWorker.

If those traces are rigth on the track, the cron worker should not do anything.

/cc @ayufan

Assignee Loading
Time tracking Loading