Replace mtail metrics with Unicorn instrumentation
Problem to solve
We currently collect a number of metrics for unicorn/sidekiq using mtail deployed on gitlab.com. This setup is fragile as it depends on external tools and parsing logs.
Further details
We deployed mtail as a stop-gap to capture metrics on usage and errors while we build direct instrumentation in unicorn. Now that the unicorn metrics code is more stable, we should eliminate this 3rd party dependency as much as possible.
Proposal
Work through all the various log line matchers in rails.mtail, sidekiq.mtail, and unicorn.mtail and convert them to internal instrumentation.
What does success look like, and how can we measure that?
We can eliminate mtail collection from production for metrics that need alerting.
Links / references
Related: https://gitlab.com/gitlab-org/gitlab-ce/issues/49214