Cleanup temp indexes from reindexing

What does this MR do?

With this change, we cleanup any "leftover" indexes from previous reindexing attempts (if any - it'll be a rare event).

During reindexing (REINDEX CONCURRENTLY), postgres creates a $index_ccnew or $index_ccnew1... index. This is meant to be temporary and upon successful creation, the index is being swapped out with the original one (and the original index is dropped). However, if the reindexing run gets aborted (cancelling REINDEX CONCURRENTLY for whatever reason), the _ccnew[0-9]*$ index sticks around. This is what we call a "leftover" index here. Upon the next reindexing run, we scan for these indexes and drop those if any.

We already introduced a rule/spec to make sure we don't have any indexes added that also match the ccnew[0-9]*$ pattern: https://gitlab.com/gitlab-org/gitlab/blob/master/spec/db/schema_spec.rb#L269

Here's how a local reindexing invocation looks like including queries. In this case, we have two leftover indexes lingering around:

abrandl:gitlab/ (ab/reindexing-cleanup✗) $ LOG_QUERIES_TO_CONSOLE=true spring rake gitlab:db:reindex                                                                                                                                                
...
D, [2021-07-14T13:51:57.607719 #1372784] DEBUG -- :   Gitlab::Database::PostgresIndex Load (6.5ms)  SELECT "postgres_indexes".* FROM "postgres_indexes" WHERE (name ~* '\_ccnew[0-9]*$') /*application:web,line:/lib/gitlab/database/reindexing.rb:26:in `cleanup_leftovers!'*/
D, [2021-07-14T13:51:57.608104 #1372784] DEBUG -- :   ↳ lib/gitlab/database/reindexing.rb:26:in `cleanup_leftovers!'
I, [2021-07-14T13:51:57.617376 #1372784]  INFO -- : Removing index public.foobar_ccnew which is a leftover, temporary index from previous reindexing activity
D, [2021-07-14T13:51:57.618798 #1372784] DEBUG -- :    (0.2ms)  SET lock_timeout TO '60000ms' /*application:web,line:/lib/gitlab/database/with_lock_retries.rb:168:in `execute'*/
D, [2021-07-14T13:51:57.619107 #1372784] DEBUG -- :   ↳ lib/gitlab/database/with_lock_retries.rb:168:in `execute'
D, [2021-07-14T13:51:57.624411 #1372784] DEBUG -- :    (4.7ms)  DROP INDEX CONCURRENTLY IF EXISTS "public"."foobar_ccnew" /*application:web,line:/lib/gitlab/database/reindexing.rb:37:in `block (3 levels) in cleanup_leftovers!'*/
D, [2021-07-14T13:51:57.624599 #1372784] DEBUG -- :   ↳ lib/gitlab/database/reindexing.rb:37:in `block (3 levels) in cleanup_leftovers!'
D, [2021-07-14T13:51:57.625226 #1372784] DEBUG -- :    (0.1ms)  RESET idle_in_transaction_session_timeout; RESET lock_timeout /*application:web,line:/lib/gitlab/database/with_lock_retries.rb:168:in `execute'*/
D, [2021-07-14T13:51:57.625398 #1372784] DEBUG -- :   ↳ lib/gitlab/database/with_lock_retries.rb:168:in `execute'
I, [2021-07-14T13:51:57.625464 #1372784]  INFO -- : Removing index public.foobar_ccnew1 which is a leftover, temporary index from previous reindexing activity
D, [2021-07-14T13:51:57.625676 #1372784] DEBUG -- :    (0.1ms)  SET lock_timeout TO '60000ms' /*application:web,line:/lib/gitlab/database/with_lock_retries.rb:168:in `execute'*/
D, [2021-07-14T13:51:57.625847 #1372784] DEBUG -- :   ↳ lib/gitlab/database/with_lock_retries.rb:168:in `execute'
D, [2021-07-14T13:51:57.629337 #1372784] DEBUG -- :    (3.0ms)  DROP INDEX CONCURRENTLY IF EXISTS "public"."foobar_ccnew1" /*application:web,line:/lib/gitlab/database/reindexing.rb:37:in `block (3 levels) in cleanup_leftovers!'*/
D, [2021-07-14T13:51:57.629915 #1372784] DEBUG -- :   ↳ lib/gitlab/database/reindexing.rb:37:in `block (3 levels) in cleanup_leftovers!'
D, [2021-07-14T13:51:57.631936 #1372784] DEBUG -- :    (0.3ms)  RESET idle_in_transaction_session_timeout; RESET lock_timeout /*application:web,line:/lib/gitlab/database/with_lock_retries.rb:168:in `execute'*/
D, [2021-07-14T13:51:57.632483 #1372784] DEBUG -- :   ↳ lib/gitlab/database/with_lock_retries.rb:168:in `execute'
D, [2021-07-14T13:51:57.733552 #1372784] DEBUG -- :   Gitlab::Database::PostgresIndex Load (91.1ms)  SELECT "postgres_indexes".* FROM "postgres_indexes" WHERE "postgres_indexes"."partitioned" = FALSE AND "postgres_indexes"."exclusion" = FALSE AND "postgres_indexes"."expression" = FALSE AND "postgres_indexes"."type" IN ('btree', 'gist') AND (name !~ '\_ccnew[0-9]*$') AND (NOT EXISTS (SELECT "postgres_reindex_actions".* FROM "postgres_reindex_actions" WHERE "postgres_reindex_actions"."state" = 1 AND (action_end > '2021-07-04 11:51:57.641293') AND (index_identifier = identifier))) AND "postgres_indexes"."ondisk_size_bytes" BETWEEN 1073741824 AND 107374182400 /*application:web,line:/lib/gitlab/database/reindexing/index_selection.rb:36:in `sort_by'*/
D, [2021-07-14T13:51:57.733895 #1372784] DEBUG -- :   ↳ lib/gitlab/database/reindexing/index_selection.rb:36:in `sort_by'

Does this MR meet the acceptance criteria?

Conformity

• I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.
• This change is backwards compatible across updates, or this does not apply.