Set maintenance mode on CustomersDot on 2021-11-13

Zuora will undergo maintenance on the 13th of November, we can put CustomersDot in maintenance mode to avoid any errors for customers.

Maintenance window

Saturday November 13, 2021 - 7:00PM to 9:00PM PDT

Process

Toggle maintenance_mode in https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags/111/edit

1. @lulalala Set maintenance mode in CustomersDot at 7PM PDT
2. @lulalala Test the maintenance page is working at 7PM PDT
3. @lulalala Quiet Sidekiq process from Sidekiq Admin UI at 7PM PDT
   • This will prevent Sidekiq from processing new queued jobs but jobs will continue to enqueue.
4. @lulalala Engage the CMOC to update status.gitlab.com
5. @lulalala Remove maintenance mode in CustomersDot at 9PM PDT
6. @lulalala Test the maintenance page is gone at 9PM PDT
7. @lulalala Manually restart Sidekiq at 9PM PDT
   • Connect to CDot production server and run sudo supervisorctl restart customers-sidekiq as documented here
8. @lulalala Engage the CMOC to update status.gitlab.com
9. @lulalala Manually enqueue scheduled jobs from the Recurring Jobs tab missed during maintenance window
   • These were likely the jobs missed based on the planned outage window:
     • ExpireOrderCronJob
     • BillingRateUpdateCronJob
     • ReconciliationCronJob
     • UpcomingReconciliationNotificationCronJob
     • AutoRenewCronJob

Alternatives

We could:

1. Automate this with a script or cron job

More details

See https://community.zuora.com/communities/community-home/digestviewer/viewthread?GroupId=475&MessageKey=68132cda-6776-4ef4-bb39-e7d389727fec for details