Replace mtail metrics with Unicorn instrumentation

Problem to solve

We currently collect a number of metrics for unicorn/sidekiq using mtail deployed on gitlab.com. This setup is fragile as it depends on external tools and parsing logs.

Further details

We deployed mtail as a stop-gap to capture metrics on usage and errors while we build direct instrumentation in unicorn. Now that the unicorn metrics code is more stable, we should eliminate this 3rd party dependency as much as possible.

Proposal

Work through all the various log line matchers in rails.mtail, sidekiq.mtail, and unicorn.mtail and convert them to internal instrumentation.

What does success look like, and how can we measure that?

We can eliminate mtail collection from production for metrics that need alerting.

Replace mtail metrics with Unicorn instrumentation

Problem to solve

Further details

Proposal

What does success look like, and how can we measure that?

Links / references