Daily members_expiring_worker sidekiq cron job on gitlab.com failing often due to SQL statement timeout

Summary

Sidekiq logs in Kibaba show the daily `embers_expiring_worker``task to notify members whose expiration dates are coming up is failing more often than not due to a query timeout in Members::ExpiringWorker`.

Steps to reproduce

Example Project

What is the current bug behavior?

Initial query of users needing to be notified fails, no notifications sent.

Users will receive late membership expiry notifications (if the job runs successfully within a week) or no notifications at all (if it fails 7 times in a row).

What is the expected correct behavior?

Job runs successfully every day.

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:env:info\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:check SANITIZE=true\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\`) (we will only investigate if the tests are passing)

Possible fixes