Monitor rails_sql apdex score for primary and secondary databases separately

Placeholder issue, spun out of #865 (closed)

The workloads and characteristics of queries hitting our primary DB and our secondary DBs are very different. As such we should monitor them independently.

At present, they are lumped together in gitlab_sql_duration_seconds_bucket. Worse, this score does not exclude Sidekiq, which drives a major portion of our primary traffic.

See also #873 (moved), which is this issue, for logs.

Solution

Solution is discussed at at #873 (comment 511091812).

• Broadcast an instrumentation events after redirection:
  • Gitlab::Database::LoadBalancing::ConnectionProxy#read_using_load_balancer broadcast replica events
  • Gitlab::Database::LoadBalancing::ConnectionProxy#write_using_load_balancer broadcast primary events
  • The ones in method missing broadcast primary events
• Dispatch prometheus metrics