Skip to content

Enable automatic database reindexing on weekends

Production Change

Change Summary

This issue tracks enabling automatic reindexing for Postgres indexes on weekends. This installs a cronjob on the deploy host, which starts reindexing processes during weekends. It uses the omnibus-supplied cookbook introduced in gitlab-org/gitlab#249662 (closed).

We chose weekends for now, since we consider this both a low-traffic and a no-deploy time currently.

The reindexing feature is currently behind a feature flag database_reindexing, which is not yet enabled. So the overall change has two steps effectively:

  1. Install cronjob, which is a no-op since the feature flag is disabled
  2. Enable feature flag and observe reindexing actions on weekends

This issue details the first of those steps.

Change Details

  1. Services Impacted - Postgres
  2. Change Technician - @abrandl
  3. Change Criticality - C4
  4. Change Type - changescheduled
  5. Change Reviewer - @nnelson
  6. Due Date - -
  7. Time tracking - Change scheduled for 2020-10-23 from 2pm UTC
  8. Downtime Component - -

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Change Steps - steps to take to execute the change

**Estimated Time to Complete (mins) - 2

Post-Change Steps - steps to take to verify the change

  • Check crond cronjob has been created in /var/opt/gitlab/crond
  • Observe reindexing on weekends, e.g. in the rails logs or through SELECT * FROM postgres_reindex_actions ORDER BY action_start DESC LIMIT 10

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Monitoring

Key metrics to observe

None

Summary of infrastructure changes

  • Does this change introduce new compute instances? No.
  • Does this change re-size any existing compute instances? No.
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No.

Summary of the above

No relevant changes.

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled).
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and resultes noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue.)
  • There are currently no active incidents.
Edited by Andreas Brandl