[Continuous Integration] Add metrics for stale build traces

We should have graphs showing how many jobs have metrics in redis.

In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3206 we used the database to check how many finished jobs still had traces in redis and noticed that there were traces that weren't migrated to object storage. The TTL on the keys is 1 week, meaning that the traces would get lost if not migrated to object storage before the TTL expires.

During the incident that caused this, the Google Cloud Storage Outage production#3188, we saw elevated error rates for the workers that are supposed to take care of uploading. But after the incident was over, things returned to normal and we didn't notice the the leftover traces until 2 days later because @cmiskell mentioned a slight increase in redis memory usage (#738 (closed)).

This is why I think we should export a gauge showing number of jobs with traces in redis labelled with their build status from the database:

SELECT COUNT("ci_builds".*), status
FROM "ci_builds"
WHERE "ci_builds"."type" = 'Ci::Build'
AND (EXISTS (
  SELECT 1 FROM "ci_build_trace_chunks" WHERE (ci_builds.id = ci_build_trace_chunks.build_id)))
group by status;
count |  status  
-------+----------
   521 | failed
  3139 | success
  6092 | running
    78 | canceled
(4 rows)

(This query might need to be optimized)

Edited by Bob Van Landuyt