[meta] Add metrics instrumentation to GitLab Internals

Need

In order to measure performance improvements for https://gitlab.com/gitlab-org/gitlab-ce/issues/26789 we need to add Prometheus metrics endpoints and instrument the code to provide stats.

Approach

There are a number of existing metrics sources and there are some gaps. We first need to identifiy the gaps.

• Identify gaps where we have no metrics to show performance.
• Add Prometheus client library support where possible.
• Replace existing metrics (ie InfluxDB) with Prometheus native methods.
• Add exporters for 3rd party software without native Prometheus support.
• Add log-based metrics generators where necessary (mtail).
• Add blackbox probing for end-to-end testing.

Benefit

Adding instrumentation with Prometheus provides us the ability to measure our performance gains, and alert on performance degradation.

Competition

Some other approaches are possible, and may be useful as temporary stop-gaps until native support is completed.

• Bridging InfluxDB data into Prometheus. 0
• Logs-based mtail metrics generators. 1

mentioned in issue #26789 (closed)

@bjk-gitlab any updates on this issue?

We're currently targeting 9.3 to implement the infrastructure to get metrics using the Prometheus Ruby client library.

https://gitlab.com/gitlab-org/gitlab-ce/issues/29118

Hi,

First of all, thank you for raising an issue to help improve the GitLab product. We're sorry about this, but this particular issue has gone unnoticed for quite some time. To establish order in the GitLab-CE Issue Tracker, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.

This issue will be closed, as it meets the following criteria:

• No activity in the past 12 months
• No milestone (unscheduled)

We'd like to ask you to help us out and determine whether this issue should be reopened.

If this issue is reporting a bug, please can you attempt to reproduce on the latest version of GitLab or GitLab.com, to help us to understand whether the bug still needs our attention.

If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.

Thanks for your help

/cc @markglenfletcher

added auto updated awaiting feedback labels

I'd consider this mostly fixed, we have pretty good performance metrics in rails now.

@andrewn, what do you think?

Yeah: there are still some improvements to be made, but I agree that the spirit of this meta-issue has been achieved :)

closed

added devopsmonitor label