Obtain correct metrics to isolate the issue with failed builds for `gitlab-com/www-gitlab-com`

Summary

Builds are sometimes failing for gitlab-com/www-gitlab-com. Examples and additional details can be found in an incident describing some of these failed builds, https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3104.

It was determined that there is an issue with our metrics making finding and correlating them for these failures difficult.

This corrective action issue is meant to enhance our metrics to allow for deeper investigation (blocks corrective action https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11942).

We have https://app.periscopedata.com/app/gitlab/561277/WIP:-www.gitlab.com-CI-CD-Pipeline-Metrics to help see the pipeline durations, but we are currently missing correlation to other events in Kibana and Grafana.

Update: Attempts to import the required data into Sisense have failed so far. See details in this thread.

There is an alternative to get some info from the Rails console, described in this thread

Edited Feb 16, 2021 by Chad Woolley