Skip to content

[gstg] - Enable LABKIT_METRICS_ENABLED

Production Staging Change

Change Summary

Relates to gitlab-com/gl-infra/observability/team#4160 (closed).

The MR introduces Labkit::Metrics, a thin wrapper around the Prometheus client. Because part of the code runs at application boot time, there's no way to use a development feature flag. For this reason, we are pushing the new implementation behind an environment variable that will be deployed disabled by default.

This CR is to enable the LABKIT_METRICS_ENABLED environment variable only in gstg. After it proves to be safe, we do the same in gprd in a new CR.

Change Details

  1. Services Impacted - ServiceWeb ServiceSidekiq ServiceAPI
  2. Change Technician - @hmerscher
  3. Change Reviewer - @knottos
  4. Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) - 2025-06-23 09:30 UTC
  5. Time tracking - ~30m
  6. Downtime Component - no downtime

Set Maintenance Mode in GitLab

If your change involves scheduled maintenance, add a step to set and unset maintenance mode per our runbooks. This will make sure SLA calculations adjust for the maintenance period.

Detailed steps for the change

Pre-execution steps

  • Make sure all tasks in Change Technician checklist are done
  • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • The SRE on-call provided approval with the eoc_approved label on the issue.
  • For C1, C2, or blocks deployments change issues, Release managers have been informed prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents that are severity1 or severity2
  • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.

Change steps - steps to take to execute the change

Estimated Time to Complete (mins) - ~30min

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - ~30min

Monitoring

Key metrics to observe

  • Metric: gitlab_service_apdex:ratio_5m, gitlab_service_errors:ratio_5m
    • Location: Platform Triage dashboard
    • The malfunctioning of the environment variable enablement could impact all metrics. Increased error rate or reduced apdex can be interpreted as signals misbehaviour.

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • The change plan is technically accurate.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • The change execution window respects the Production Change Lock periods.
  • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
  • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue. Mention @gitlab-org/saas-platforms/inframanagers in this issue to request approval and provide visibility to all infrastructure managers.
  • For C1, C2, or blocks deployments change issues, confirm with Release managers that the change does not overlap or hinder any release process (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
Edited by Hercules Merscher