Establish migration strategy for repository cache

Previously in &857, MultiStore would perform dual-reads and report differences to a metric. The migration strategy was to observe the fall in differences until it reaches 0 (refer to #2052 (closed)).

The simple strategy would be to let the application run in dual-write phase for a long as the longest TTL which is 2 weeks (https://gitlab.com/gitlab-org/gitlab/-/blob/b87a6a83a6f55c7029cda014c42f74af594a2ee6/lib/gitlab/repository_set_cache.rb#L8). Other TTLs to note

Example of a key for such a set of data

redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[2]> ttl cache:gitlab:tag_names:gitlab-org/gitlab-shell:3:set
(integer) 1124614
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[2]> type cache:gitlab:tag_names:gitlab-org/gitlab-shell:3:set
set

To speed things up, we could run the external migration script https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/redis_diff.rb scoped to set keys.

Proposed plan

  1. Start dual-write using feature-flags and MultiStore
  2. Wait >1 day for hash cache and regular cache to sync up
  3. Run redis external validator script with --pattern=cache:gitlab:*:set.
Edited by Sylvester Chin