Skip to content

Add queuing mechanic for reindexing

Andreas Brandl requested to merge ab/reindexing-queue into master

What does this MR do and why?

This adds a queue for reindexing operations. The idea is that we can specifically schedule indexes for reindexing (which typically happens on the weekends), even if we don't support bloat estimation for those (e.g. for GIN) or if they wouldn't be picked up by the bloat heuristic.

So the overall process looks like so:

  1. Trigger async index creation if necessary (still a hack)
  2. Cleanup leftovers from previous run
  3. Consume from the queue if non-empty
  4. Execute heuristic to choose indexes for regular reindexing

Relates to #330922 (closed)

How to set up and validate locally

Feature.enable(:database_reindexing)
  1. Start with an empty database
  2. Enqueue an index: spring rake "gitlab:db:enqueue_reindexing_action[public.index_ci_runners_on_description_trigram]"
  3. Run reindexing (it'll pick up 2 indexes per run): LOG_QUERIES_TO_CONSOLE=true bundle exec rake gitlab:db:reindex
  4. Examine the queue: select * from postgres_reindex_queued_actions;
  5. Examine the recorded actions: select * from postgres_reindex_actions;

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Andreas Brandl

Merge request reports