Record application server type in Prometheus

• Refs gitlab#219114 (closed)
• Client side MR: gitlab!35618 (merged)

For Usage Ping we have been collecting the app_server_type for a while now, but that data is always wrong, because it is evaluated based on the client runtime, which will always be a sidekiq worker, not a Rails app server.

In order to reliably know which app server (Puma or Unicorn) is running on which node in any given Omnibus installation, we need to push this logic out of the application runtime and down to Prometheus. Fortunately, we sort of already have this data: since both Unicorn and Puma export metrics to Prometheus, the mere presence of these will tell us what is running.

To implement this, I am adding two recording rules here that record a new metric called gitlab_usage_ping:node_service_app_server_workers:sum, which will either produce the number of workers if metrics of the respective server type are found, or no results. Sample output:

gitlab_usage_ping:node_service_app_server_workers:sum{instance="web:8080", job="gitlab-rails", server="unicorn"}    2
gitlab_usage_ping:node_service_app_server_workers:sum{instance="web:8080", job="gitlab-rails", server="puma"}    4

The client can then simply query for gitlab_usage_ping:node_service_app_server_workers:sum and can check the server label to know which app server is running on the given instance/job combination.