Skip to content

Add background migration worker for Redis

Sylvester Chin requested to merge sc1-background-migrate-redis into master

What does this MR do and why?

This MR introduces BackfillProjectPipelineStatusTtlWorker and BackgroundMigration::RedisWorker which performs background migrations for Redis.

Clears 2nd requirement for #416007 (closed)

Part of gitlab-com/gl-infra/scalability#2401 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

On gdk rails c,

  1. Apply this branch via gdk restart rails

  2. Setup some dummy data on rails console

[1] pry(main)> redis = Redis.new(Gitlab::Redis::Cache.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/2>
[2] pry(main)> (0..10000).each do |i|
[2] pry(main)*   redis.set("cache:gitlab:project:#{i}:pipeline_status", i)
[2] pry(main)* end
  1. Run the worker
RedisMigrationWorker.perform_async('BackfillProjectPipelineStatusTtl', '0')
  1. Check the ttl on the keys (should be a positive number) and its eventual deletion
ttl cache:gitlab:project:131:pipeline_status # or any other number between 1 to 10000

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports