Add Redis support in background migrations
The current mode for background migration is database-focused (https://docs.gitlab.com/ee/development/database/background_migrations.html and https://docs.gitlab.com/ee/development/database/batched_background_migrations.html). With greater adoption of Redis Cluster, we could likely start seeing more use cases for Redis-based background migration like in gitlab-org/gitlab#416007 (closed).
One idea is a cursor-based migration since Redis uses scan
to search/filter for keys. However, batched background migration is built for database-specific migration. We are better off doing Redis background migrations in a more lightweight manner since there are fewer parameters to work with:
- key pattern to scan for
- batch size
- scan size
- cursor
- logic to apply on the scanned keys
Some extra info:
-
SCAN
provides some useful guarantees: (1) a full iteration will return all keys that were present at the start of the iteration (provided they were not deleted) and (2) a full iteration never returns a key NOT present at the start - https://github.com/redis/redis/blob/9600553ef2273411beb6168d44af70328f971eb8/src/dict.c#L1217 more detailed explanation on how such guarantees are possible
Proposal
We can leverage Sidekiq to run the redis background migration. Within each Sidekiq job, we run the logic up to the batch size and trigger a subsequent job to avoid choking up the worker on 1 huge long-running job.
Example workflow
- PDM runs and triggers
BackfillTtlWorker.perform_async(cursor=0, batch_size=50000, scan_size=1000)
- The job gets picked up by a Sidekiq pod
- The
BackfillTtlWorker
runs 50 scans and invokesBackfillTtlWorker.perform_async(cursor=<updated_cursor>, batch_size=50000, scan_size=1000)
- The worker terminates when
cursor
reaches 0.
Suggestion (at the risk of over-designing): We can probably create a RedisMigrationHelper
to abstract away the background running logic. BackfillTtlWorker
only needs to contain the logic of dealing with a set of keys.