Setting up Redis MultiStore via config

Overview

This issue is spawned from #2059 (closed) as a follow up to the discussions in gitlab-org/gitlab!106123 (comment 1209290884). Sharding/migrating with Redis introduces unnecessary toil as most of the steps involved are copy-and-pasting MultiStore code. See gitlab-org/gitlab!106123 (merged) for example on the amount of work needed.

Objective: Discuss and decide on how multistore should be defined in config

Proposed Format

Based on gitlab-org/gitlab!106123 (comment 1209379747). It introduces a new key: multi_store.

<env>:
  <store name>:
    multi_store:
      <primary/secondary>: "" # option 1, string pointing to another declaration
      <primary/secondary>: # option 2, define store details within multi_store
        url: ""
        db: 1

New keys

1. multi_store: Define primary and secondary store names or connection details.

Example usage

Continuing the example cited by @jacobvosmaer-gitlab in gitlab-org/gitlab!106123 (comment 1209379747)

Step 1. Define Gitlab::Redis::RepositoryCache and update desired call sites with new singleton. Note that this only applies when sharding out a fraction of keys from an existing instance. For full migration, just the config change would suffice.

Step 2. Define Multistore-as-config

With the config below, all call sites using Gitlab::Redis::RepositoryCache now uses MultiStore as its redis instead. With feature-flags defaulting to false, all operations route to the secondary store (cache) initially. Double-writes and cutover can be controlled via feature-flags like we do now.

`config/redis.yml`
---
production:
  cache:
    url: xxx
    sentinels: [...]
  repository_cache:
    multi_store:
      primary:
        url: xxx
        sentinels: [...]
      secondary: cache // must point to the existing store

As shown above, a multi_store key needs to be defined with a valid primary and secondary store. Instrumentation classes for stores can be generated on initialization (https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/instrumentation/redis.rb#L12 may need to include classes defined in config but not in code). For example, repository store details will be used to dynamically generate the primary store without having to introduce a RepositoryCacheMigrator singleton like in gitlab-org/gitlab!106123 (merged).

Note that config can be updated to define new MultiStores without feature-flag definition yamls since we only raise errors for missing definition files in test and development.