Enable automatic database reindexing on weekends
Production Change
Change Summary
This issue tracks enabling automatic reindexing for Postgres indexes on weekends. This installs a cronjob on the deploy host, which starts reindexing processes during weekends. It uses the omnibus-supplied cookbook introduced in gitlab-org/gitlab#249662 (closed).
We chose weekends for now, since we consider this both a low-traffic and a no-deploy time currently.
The reindexing feature is currently behind a feature flag database_reindexing
, which is not yet enabled. So the overall change has two steps effectively:
- Install cronjob, which is a no-op since the feature flag is disabled
- Enable feature flag and observe reindexing actions on weekends
This issue details the first of those steps.
Change Details
- Services Impacted - Postgres
- Change Technician - @abrandl
- Change Criticality - C4
- Change Type - changescheduled
- Change Reviewer - @nnelson
- Due Date - -
- Time tracking - Change scheduled for 2020-10-23 from 2pm UTC
- Downtime Component - -
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Change Steps - steps to take to execute the change
**Estimated Time to Complete (mins) - 2
-
Apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4425 to gprd -
Enable feature database_reindexing
for production
Post-Change Steps - steps to take to verify the change
-
Check crond
cronjob has been created in/var/opt/gitlab/crond
-
Observe reindexing on weekends, e.g. in the rails logs or through SELECT * FROM postgres_reindex_actions ORDER BY action_start DESC LIMIT 10
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
-
Revert: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4425 -
Revert: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4306 -
Apply reverts to gprd -
Check cronjob in /var/opt/gitlab/crond
is gone
Monitoring
Key metrics to observe
None
Summary of infrastructure changes
-
Does this change introduce new compute instances? No. -
Does this change re-size any existing compute instances? No. -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No.
Summary of the above
No relevant changes.
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Andreas Brandl