Manually queued reindex actions are lost if excllusive lease is not obtained.

Summary

When processing manually queued reindex actions, if we can not obtain exclusive lease, we just mark the action as done and never try do it again. See https://gitlab.slack.com/archives/CNZ8E900G/p1671406955830819?thread_ts=1671116341.455659&cid=CNZ8E900G (internal) for example.

Steps to reproduce

Queue new reindex action for ci database:

$ bundle exec rails gitlab:db:enqueue_reindexing_action['public.index_ci_builds_on_token_encrypted','ci'] 
Queued reindexing action: queued action [ id = 4, index: public.index_ci_builds_on_token_encrypted ]
There are 1 queued actions in total.

This creates a record in postgres_reindex_actions:

gitlabhq_development_ci=# table postgres_reindex_actions;
 id | action_start | action_end | ondisk_size_bytes_start | ondisk_size_bytes_end | state | index_identifier | bloat_estimate_bytes_start 
----+--------------+------------+-------------------------+-----------------------+-------+------------------+----------------------------
(0 rows)

Time: 1.025 ms
gitlabhq_development_ci=# table postgres_reindex_queued_actions;
 id |             index_identifier              | state |          created_at           |          updated_at           
----+-------------------------------------------+-------+-------------------------------+-------------------------------
  4 | public.index_ci_builds_on_token_encrypted |     0 | 2022-12-20 00:18:18.034824+00 | 2022-12-20 00:18:18.034824+00
(1 row)

In Rails console obtain exclusive lease using the key for main:

[1] pry(main)> lease = Gitlab::ExclusiveLease.new('gitlab/database/reindexing/coordinator/main', timeout: 5.minutes)
=> #<Gitlab::ExclusiveLease:0x000000015eb34ea8 @redis_shared_state_key="gitlab:exclusive_lease:gitlab/database/reindexing/coordinator/main", @timeout=5 minutes, @uuid="f3ede769-b425-4d1b-98e3-bc391d454326">
[2] pry(main)> lease.try_obtain
=> "f3ede769-b425-4d1b-98e3-bc391d454326"

Run the reindex task:

$ bundle exec rails gitlab:db:reindex['ci'] 

The record in postgres_reindex_queued_actions is marked as done (state = 1) but there is no corresponding record in postgres_reindex_actions:

gitlabhq_development_ci=# table postgres_reindex_queued_actions;
 id |             index_identifier              | state |          created_at           |          updated_at           
----+-------------------------------------------+-------+-------------------------------+-------------------------------
  4 | public.index_ci_builds_on_token_encrypted |     1 | 2022-12-20 00:18:18.034824+00 | 2022-12-20 00:22:03.420682+00
(1 row)

Time: 0.418 ms
gitlabhq_development_ci=# table postgres_reindex_actions;
 id | action_start | action_end | ondisk_size_bytes_start | ondisk_size_bytes_end | state | index_identifier | bloat_estimate_bytes_start 
----+--------------+------------+-------------------------+-----------------------+-------+------------------+----------------------------
(0 rows)

What is the current bug behavior?

If exclusive lease is not obtained we mark the queued reindex action as done, even if it was not executed.

What is the expected correct behavior?

We should not mark the reindex action as done, so that it can be retried on the next run.

Possible fixes

Mark the action as done only of the exclusive lease was obtained.