Skip to content

[gstg] Move repository workload from redis-cache to redis-repository-cache

To move over repository traffic:

  1. Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!2499 (merged) and https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/2759 to provide connection information for redis-repository-cache in helm charts.
  2. Update chef-vault with correct password:
We also need to update chef-vault with the correct password:
stejacks@atlas~/projects/gitlab-com/gl-infra/chef-repo (master) $ EDITOR=vim bin/gkms-vault-edit gitlab-omnibus-secrets gstg

"gitlab-rails": {
...
 "redis_repository_cache_instance": "redis://<IAMAPASSWORD>@gstg-redis-repository-cache",
...
  1. Start double writes:
# start double write
/chatops run feature set use_primary_and_secondary_stores_for_repository_cache true --staging --staging-ref

# to check
/chatops run feature get use_primary_and_secondary_stores_for_repository_cache --staging --staging-ref

# rollback
/chatops run feature set use_primary_and_secondary_stores_for_repository_cache false --staging --staging-ref
  1. Confirm keys are going to both places.

https://dashboards.gitlab.net/d/redis-repository-cache-main/redis-repository-cache-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg

https://dashboards.gitlab.net/d/redis-cache-main/redis-cache-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg

  1. Sit around for a certain amount of time (not very long in staging, I suspect) and then...

  2. Switch writes 100% to the new location.

# start primary write
/chatops run feature set use_primary_store_as_default_for_repository_cache true --staging --staging-ref

# to check
/chatops run feature get use_primary_store_as_default_for_repository_cache --staging --staging-ref

# Turn off dual write

/chatops run feature set use_primary_and_secondary_stores_for_repository_cache false --staging --staging-ref

# rollback
/chatops run feature set use_primary_store_as_default_for_repository_cache false --staging --staging-ref
  1. Check traffic drop for redis-cache out of curiosity and record in this issue.
Edited by Stephanie Jackson