[gstg] Create GIN monitoring function
Staging
Change
Change Summary
[gstg]
Create GIN monitoring function.
A new function must be created for monitoring the generalized inverted index (GIN) meta pages size.
Base for the MR: gitlab-cookbooks/gitlab-exporters!244 (merged)
Issue related: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14386
Change Details
- Services Impacted - ServicePostgres
- Change Technician - @nnelson
- Change Reviewer - @Finotto
-
Time tracking -
30 minutes
-
Downtime Component -
No downtime
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete - 5 minutes
-
Establish a secure shell session the production leader patroni system: ssh patroni-01-db-gstg.c.gitlab-staging-1.internal
. -
Clone the gitlab-com/gl-infra/production
project:rm -rf /tmp/production git clone https://gitlab.com/gitlab-com/gl-infra/production.git /tmp/production
Change Steps - steps to take to execute the change
Estimated Time to Complete - 5 minutes
-
Set label changein-progress on this issue -
To create the function, execute: sudo /tmp/production/src/gl-infra-5849-create-postgres_gin_pending_list_size-function/create-postgres_gin_pending_list_size-function.sh
-
To grant execute permissions for the function, execute: sudo /tmp/production/src/gl-infra-5849-create-postgres_gin_pending_list_size-function/grant-execute-to-postgres_gin_pending_list_size-function.sh
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete - 2 minutes
-
To verify that the function is executable as the postgres_exporter
user, execute:sudo gitlab-psql --username=postgres_exporter --command='SELECT index_name AS index_name, pending_pages AS pending_pages FROM postgres_gin_pending_list_size();'
-
Clean up the production support scripting: rm -rf /tmp/production
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete - 5 minutes
-
To revoke execute permissions for the function, execute: sudo /tmp/production/src/gl-infra-5849-create-postgres_gin_pending_list_size-function/revoke-execute-from-postgres_gin_pending_list_size-function.sh
-
To drop the function, execute: sudo /tmp/production/src/gl-infra-5849-create-postgres_gin_pending_list_size-function/drop-postgres_gin_pending_list_size-function.sh
Monitoring
Key metrics to observe
- Metric:
patroni Service Apdex
- Location: grafana
- What changes to this metric should prompt a rollback: Any decrease in Service Apdex of more than
0.5%
for longer than a few minutes.
- Metric:
patroni Service Error Ratio
- Location: grafana
- What changes to this metric should prompt a rollback: Any increase in error ratio more than
0.6%
for longer than a few minutes.
Summary of infrastructure changes
-
Does this change introduce new compute instances? No
-
Does this change re-size any existing compute instances? No
-
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by Nels Nelson