Skip to content

Add db_config_name label to prometheus database metrics

Dylan Griffith requested to merge 332945-prometheus-label-db-config-name into master

What does this MR do?

The sharding team is adding support for multiple databases and in order for us to have good visibility into performance we'll want to be able to distinguish between the different databases in our prometheus metrics.

Adding a prometheus label allows us to filter/group by the different databases.

By default there will only be 1 db_config_name which is primary by default. But this is configured by config/database.yml and in future there will be 2 databases called main and ci.

This MR also ensures the new label is disabled by default to allow us to introduce this to production in a slightly more controlled way. We opted to use an environment variable instead of feature flag in !66885 (merged) because it's too expensive to query the databases so frequently to check the feature flag value.

For now this does not increase the cardinality of these metrics as our production environment is configured with only 1 db_config_name. When we enable multiple databases in production it will double the cardinality.

Screenshots or Screencasts (strongly suggested)

$ curl http://127.0.0.1:3000/-/metrics | grep db_config_name
...
gitlab_sql_replica_duration_seconds_bucket{controller="",action="",feature_category="",db_config_name="primary",le="0.1"} 25
gitlab_sql_replica_duration_seconds_bucket{controller="",action="",feature_category="",db_config_name="primary",le="0.25"} 25
...
gitlab_sql_replica_duration_seconds_sum{controller="ProjectsController",action="show",feature_category="projects",db_config_name="primary"} 0.28392653700393566
...
gitlab_transaction_db_replica_count_total{controller="",action="",feature_category="",db_config_name="primary"} 25
gitlab_transaction_db_write_count_total{controller="ProjectsController",action="show",feature_category="projects",db_config_name="primary"} 3

How to setup and validate locally (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Closes #332945 (closed)

Edited by Dylan Griffith

Merge request reports