[gstg] Move lb workload from redis to redis-db-load-balancing

To move over repository traffic:

• Merge https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/2873 and gitlab-com/gl-infra/k8s-workloads/gitlab-com!2552 (merged) to provide connection information for redis-db-load-balancing in chef and k8s.
• Update chef-vault with correct password:

bin/gkms-vault-edit gitlab-omnibus-secrets gstg

"gitlab-rails": {
...
  "redis_yml_override": {
    "db_load_balancing": {
      "url": "redis://<IAMAPASSWORD>@gstg-redis-db-load-balancing"
    }
  },
...

• Start double writes:

# start double write
/chatops run feature set use_primary_and_secondary_stores_for_db_load_balancing true --staging --staging-ref

# to check
/chatops run feature get use_primary_and_secondary_stores_for_db_load_balancing --staging --staging-ref

# rollback
/chatops run feature set use_primary_and_secondary_stores_for_db_load_balancing false --staging --staging-ref

• Check the multi-store error logs: https://nonprod-log.gitlab.net/goto/a973cd90-abc1-11ed-9af2-6131f0ee4ce6

• Monitor the Prometheus metrics gitlab_redis_multi_store_pipelined_diff_error_total, and gitlab_redis_multi_store_read_fallback_total. They should converge to zero. Link to Thanos

• Switch writes 100% to the new location.

# start primary write
/chatops run feature set use_primary_store_as_default_for_db_load_balancing true --staging --staging-ref

# to check
/chatops run feature get use_primary_store_as_default_for_db_load_balancing --staging --staging-ref

# Turn off dual write

/chatops run feature set use_primary_and_secondary_stores_for_db_load_balancing false --staging --staging-ref

# rollback
/chatops run feature set use_primary_store_as_default_for_db_load_balancing false --staging --staging-ref

• Check the multi-store error logs: https://nonprod-log.gitlab.net/goto/a973cd90-abc1-11ed-9af2-6131f0ee4ce6

• Monitor the Prometheus metrics gitlab_redis_multi_store_pipelined_diff_error_total, gitlab_redis_multi_store_read_fallback_total, and gitlab_redis_multi_store_method_missing_total. They should all be zero. Link to Thanos