Add fallback logic to instrumentation classes

What does this MR do and why?

Service Ping instrumentation classes framework being open to extension by anybody who wants to track some Product Intelligence data is prone to bugs and implementation errors. However it should be resilient as a process and not fail due to single metric being broken. To achieve that fallback mechanism is required, that would rescue any error coming form single metric layer

Recently there were multiple problems that could be avoided if fallback mechanism was in place:

1. https://gitlab.com/gitlab-org/gitlab/-/issues/362743#note_1016983910
2. #365437 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

1. Renamce any metric instrumentation class eg: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/usage/metrics/instrumentations/count_slack_app_installations_gbp_metric.rb

$ mv ee/lib/gitlab/usage/metrics/instrumentations/count_slack_app_installations_gbp_metric.rb ee/lib/gitlab/usage/metrics/instrumentations/count_slack_app_installations_gbp_metric.rb.old

For development env:

Open Rails console in production mode

$ bin/rails console

In rails console build ServicePing payload and check that metric error has been raised

  [1] pry(main)> payload = ServicePing::BuildPayload.new.execute
  NameError: uninitialized constant Gitlab::Usage::Metrics::Instrumentations::CountSlackAppInstallationsGbpMetric
Did you mean?  Gitlab::Usage::Metrics::Instrumentations::CountSlackAppInstallationsMetric
from /Users/mikolajwawrzyniak/.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.7/lib/active_support/inflector/methods.rb:288:in `const_get'

For production env:

Stub production env configuration

diff --git a/config/database.yml b/config/database.yml
index 78cde3996c6..3fe30cdd95f 100644
+++ a/config/database.yml
--- b/config/database.yml
@@ -1,25 +1,3 @@
+production:
+  main:
+  adapter: postgresql
+    encoding: unicode
+    database: gitlabhq_development
+    host: /Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/postgresql
+    port: 5432
+    pool: 10
+    prepared_statements: false
+    variables:
+      statement_timeout: 120s
+  ci:
+    adapter: postgresql
+    encoding: unicode
+    database: gitlabhq_development_ci
+    host: /Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/postgresql
+    port: 5432
+    pool: 10
+    prepared_statements: false
+    variables:
+      statement_timeout: 120s
+
development:
  main:
  adapter: postgresql

diff --git a/config/redis.cache.yml b/config/redis.cache.yml
index 43251d6908c..4da3d3ba0e1 100644
+++ a/config/redis.cache.yml
--- b/config/redis.cache.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=4"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=4"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=14"
diff --git a/config/redis.queues.yml b/config/redis.queues.yml
index ed05c84358a..9bc817513ee 100644
+++ a/config/redis.queues.yml
--- b/config/redis.queues.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=1"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=1"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=11"
diff --git a/config/redis.rate_limiting.yml b/config/redis.rate_limiting.yml
index 43251d6908c..4da3d3ba0e1 100644
+++ a/config/redis.rate_limiting.yml
--- b/config/redis.rate_limiting.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=4"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=4"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=14"
diff --git a/config/redis.sessions.yml b/config/redis.sessions.yml
index 24c7b947c95..6c26ddd8cf7 100644
+++ a/config/redis.sessions.yml
--- b/config/redis.sessions.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=5"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=5"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=15"
diff --git a/config/redis.shared_state.yml b/config/redis.shared_state.yml
index d1073a8b17f..aef8c13bb55 100644
+++ a/config/redis.shared_state.yml
--- b/config/redis.shared_state.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=0"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=0"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=10"
diff --git a/config/redis.trace_chunks.yml b/config/redis.trace_chunks.yml
index 807631fd1fa..0744da9950b 100644
+++ a/config/redis.trace_chunks.yml
--- b/config/redis.trace_chunks.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=3"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=3"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=13"
diff --git a/config/resque.yml b/config/resque.yml
index d1073a8b17f..aef8c13bb55 100644
+++ a/config/resque.yml
--- b/config/resque.yml
@@ -1,3 +1,2 @@
+production: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=0"
 development: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=0"
 test: "unix:/Users/mikolajwawrzyniak/gitlab/gitlab-development-kit/redis/redis.socket?db=10"

Open Rails console in production mode

$ bin/rails console -e production

In rails console in production mode build ServicePing payload and check that metric value has be replaced with fallback value -1 and no error has been raised

  irb(main):001:0> payload = ServicePing::BuildPayload.new.execute
  irb(main):002:0> payload["counts"]["slack_app_installations_gbp"]
  => -1

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.