2026-08-12: [grpd] Scaleout redis-cluster-cache by resharding the keys to new shard
<!--Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies.-->
# Production Change
## Change Summary
We are horizontally scaling the production (`gprd`) `redis-cluster-cache` by moving keyslots from the existing five masters onto the newly added shard-06 master. The cluster will be balanced across six masters at approximately 2,731 slots per master.
Prerequisite: the shard-06 nodes must be provisioned and joined to the cluster through https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22601. The shard-06 master must be healthy and own zero slots before this change begins.
The current five-master distribution should be four masters with 3,277 slots and one master with 3,276 slots. This change moves 546 slots from each existing master:
- `5 × 546 = 2,730` slots moved to shard 06.
- Existing 3,277-slot masters will finish with 2,731 slots.
- The existing 3,276-slot master will finish with 2,730 slots.
- Shard 06 will finish with 2,730 slots.
The final six-master distribution should therefore be four masters with 2,731 slots and two masters with 2,730 slots.
This procedure was validated in staging through https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22613.
## Change Details
<!--To automatically add your change to the GitLab Production calendar update the following fields:
- Time tracking
- Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)
Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler-->
1. **Services Impacted** - ~"Service::RedisClusterCache"
2. **Change Technician** - `@roshs`
3. **Change Reviewer** - @fshabir
4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-18 12:00
5. **Time tracking** - _60 minutes_
6. **Downtime Component** - No downtime expected
> [!note]
>
> To find a **Change Reviewer**, use the [Reviewer Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette/?currentProject=production&order=1) scoped to the `production` project. Spin the wheel. If roulette selects the current EOC (check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW)), please re-spin to select another reviewer.
> [!important]
>
> If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period.
## Preparation
> [!note]
>
> The following checklists must be done in advance, before setting the label ~"change::scheduled"
### Change Reviewer checklist
<!--To be filled out by the reviewer.-->
~C4 ~C3 ~C2 ~C1:
- [x] Check if the following applies:
- The **scheduled day and time** of execution of the change is appropriate.
- The [change plan](#detailed-steps-for-the-change) is technically accurate.
- The change plan includes **estimated timing values** based on previous testing.
- The change plan includes a viable [rollback plan](#rollback).
- The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change.
~C2 ~C1:
- [x] Check if the following applies:
- The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
- The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
- The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary.
### Change Technician checklist
- [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed.
- [x] The [change plan](#detailed-steps-for-the-change) is technically accurate.
- [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access.
- [x] This Change Issue is linked to the appropriate Issue and/or Epic
- [x] Change has been tested in staging and results noted in a comment on this issue.
- [ ] A dry-run has been conducted and results noted in a comment on this issue.
- [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl).
- [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"`
- [x] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours.
- [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers.
- [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.)
- [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time.
## Detailed steps for the change
### Pre-execution steps
> [!note]
>
> The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed below.
- [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done
- [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out.
- [x] The SRE on-call provided approval with the ~eoc_approved label on the issue.
- [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.)
- [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2
- [x] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
- [x] Confirm the production node-add CR is complete: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22601
- [x] Confirm all 18 Redis nodes are healthy.
- [x] Confirm shard 06 has one master, two healthy replicas, and zero assigned slots.
- [x] Confirm the cluster has five existing slot-owning masters.
- [x] Confirm all 16,384 slots are assigned and healthy.
- [x] Confirm `cluster_slots_fail:0` and `cluster_slots_pfail:0`.
- [x] Open the production Redis monitoring dashboards before starting.
### Change steps - steps to take to execute the change
_Estimated Time to Complete (mins)_ - 90
- [x] Set label ~"change::in-progress" \`/label
- [x] Confirm the prerequisite [node-add CR](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22601) is complete: shard-06 nodes joined, the shard-06 master is healthy, and it owns **0 slots**.
- [x] Capture the current master node IDs and the new shard-06 master ID. **Revalidate at execution time because a failover changes which node in a shard is the master.**
```shell
ssh redis-cluster-cache-shard-01-01-db-gprd.c.gitlab-production.internal
# List current masters:
# 5 masters with slot ranges, plus the shard-06 master with 0 slots.
NODES=$(sudo gitlab-redis-cli cluster nodes | grep master)
echo "$NODES"
# Export the 5 existing masters' IDs.
export M1=$(echo "$NODES" | awk '$2 ~ /shard-01-/ {print $1}')
export M2=$(echo "$NODES" | awk '$2 ~ /shard-02-/ {print $1}')
export M3=$(echo "$NODES" | awk '$2 ~ /shard-03-/ {print $1}')
export M4=$(echo "$NODES" | awk '$2 ~ /shard-04-/ {print $1}')
export M5=$(echo "$NODES" | awk '$2 ~ /shard-05-/ {print $1}')
# Export the current shard-06 master ID.
export SHARD06_MASTER_ID=$(echo "$NODES" | awk '$2 ~ /shard-06-/ {print $1}')
# Validate variable assignment.
for pair in \
"M1=$M1" \
"M2=$M2" \
"M3=$M3" \
"M4=$M4" \
"M5=$M5" \
"SHARD06_MASTER_ID=$SHARD06_MASTER_ID"
do
name=${pair%%=*}
val=${pair#*=}
if printf '%s' "$val" | grep -qE '^[0-9a-f]{40}$'; then
echo "OK: $name = $val"
else
echo "ERROR: $name is not exactly one node ID: [$val]"
fi
done
sudo gitlab-redis-cli cluster nodes | grep "$SHARD06_MASTER_ID"
# Expect: shard-06 node flagged "master" with no slot range.
```
- [x] Confirm the starting cluster state and slot distribution:
```shell
sudo gitlab-redis-cli cluster info |
grep -E 'cluster_state|cluster_slots_ok|cluster_slots_pfail|cluster_slots_fail|cluster_known_nodes'
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
```
Expected:
- [x] `cluster_state:ok`
* `cluster_slots_ok:16384`
* `cluster_slots_pfail:0`
* `cluster_slots_fail:0`
* `cluster_known_nodes:18`
* All 16,384 slots are covered.
* All nodes agree about the slot configuration.
* There are 5 slot-owning masters.
* The shard-06 master owns 0 slots.
- [x] **Batch 1**: move 546 slots from master 1, starting conservatively at `--cluster-pipeline 50`. Observe CPU, memory, error ratio, operation rate, failed slots, and redirection metrics. Interrupt with Ctrl-C if it competes with user operations.
```shell
time sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$M1" \
--cluster-to "$SHARD06_MASTER_ID" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
```
- [x] Verify cluster health and service metrics before continuing:
```shell
sudo gitlab-redis-cli cluster info |
grep -E 'cluster_state|cluster_slots_ok|cluster_slots_pfail|cluster_slots_fail'
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
```
- [ ] Optional: if an interrupted migration leaves open or inconsistent slots, inspect the cluster and repair it before continuing:
```shell
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
sudo gitlab-redis-cli --cluster fix 127.0.0.1:6379
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
```
Do not start another batch until all slots are covered, failed slots are zero, all nodes agree, and service metrics are healthy.
- [x] **Batch 2**: move 546 slots from master 2. The pipeline can increase to 100 if batch 1 completed cleanly. Observe metrics.
```shell
time sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$M2" \
--cluster-to "$SHARD06_MASTER_ID" \
--cluster-slots 546 \
--cluster-pipeline 100 \
--cluster-yes
```
- [x] Verify cluster health and service metrics before continuing.
- [x] **Batch 3**: move 546 slots from master 3. Observe metrics.
```shell
time sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$M3" \
--cluster-to "$SHARD06_MASTER_ID" \
--cluster-slots 546 \
--cluster-pipeline 100 \
--cluster-yes
```
- [x] Verify cluster health and service metrics before continuing.
- [x] **Batch 4**: move 546 slots from master 4. Observe metrics.
```shell
time sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$M4" \
--cluster-to "$SHARD06_MASTER_ID" \
--cluster-slots 546 \
--cluster-pipeline 100 \
--cluster-yes
```
- [x] Verify cluster health and service metrics before continuing.
- [x] **Batch 5**: move 546 slots from master 5. Observe metrics.
```shell
time sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$M5" \
--cluster-to "$SHARD06_MASTER_ID" \
--cluster-slots 546 \
--cluster-pipeline 100 \
--cluster-yes
```
- [x] Recheck that all 16,384 slots are covered and the 6 masters are balanced at approximately 2,731 slots each:
```shell
sudo gitlab-redis-cli cluster info |
grep -E 'cluster_state|cluster_slots_assigned|cluster_slots_ok|cluster_slots_pfail|cluster_slots_fail|cluster_known_nodes'
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
sudo gitlab-redis-cli cluster nodes | grep master
```
Expected:
- [x] `cluster_state:ok`
* `cluster_slots_assigned:16384`
* `cluster_slots_ok:16384`
* `cluster_slots_pfail:0`
* `cluster_slots_fail:0`
* `cluster_known_nodes:18`
* All 16,384 slots are covered.
* All nodes agree about the slot configuration.
* Four masters own 2731 slots.
* Two masters own 2730 slots.
* The shard-06 master owns 2730 slots.
* Redis service metrics remain healthy.
- [ ] Set label ~"change::complete": `/label ~change::complete`
## Rollback
### Rollback steps - steps to be taken in the event of a need to rollback this change
_Estimated Time to Complete (mins)_ - 60
> [!note]
>
> `SHARD06_MASTER_ID` and `M1..M5` must be exported in the shell before running rollback. Revalidate them at rollback time because a failover can change the current master for a shard.
>
> Each rollback batch moves slots **from** the shard-06 master back **to** the master associated with the corresponding forward batch, in reverse order (`5 → 1`).
>
> **Partial migration:** if the change is aborted before all 5 forward batches complete, only run rollback for the batches that completed. Check the shard-06 master's current slot count before starting rollback. Do not run additional reverse batches after shard-06 reaches zero slots.
- [ ] Connect to an existing production cluster node:
```shell
ssh redis-cluster-cache-shard-01-01-db-gprd.c.gitlab-production.internal
```
- [ ] Revalidate the current master IDs:
```shell
NODES=$(sudo gitlab-redis-cli cluster nodes | grep master)
echo "$NODES"
export M1=$(echo "$NODES" | awk '$2 ~ /shard-01-/ {print $1}')
export M2=$(echo "$NODES" | awk '$2 ~ /shard-02-/ {print $1}')
export M3=$(echo "$NODES" | awk '$2 ~ /shard-03-/ {print $1}')
export M4=$(echo "$NODES" | awk '$2 ~ /shard-04-/ {print $1}')
export M5=$(echo "$NODES" | awk '$2 ~ /shard-05-/ {print $1}')
export SHARD06_MASTER_ID=$(echo "$NODES" | awk '$2 ~ /shard-06-/ {print $1}')
for pair in \
"M1=$M1" \
"M2=$M2" \
"M3=$M3" \
"M4=$M4" \
"M5=$M5" \
"SHARD06_MASTER_ID=$SHARD06_MASTER_ID"
do
name=${pair%%=*}
val=${pair#*=}
if printf '%s' "$val" | grep -qE '^[0-9a-f]{40}$'; then
echo "OK: $name = $val"
else
echo "ERROR: $name is not exactly one node ID: [$val]"
fi
done
sudo gitlab-redis-cli cluster nodes | grep "$SHARD06_MASTER_ID"
```
- [ ] Run migrations in reverse order. Run only the batches corresponding to completed forward migrations:
```shell
# Rollback batch 5: shard-06 -> master 5
sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$SHARD06_MASTER_ID" \
--cluster-to "$M5" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
# Rollback batch 4: shard-06 -> master 4
sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$SHARD06_MASTER_ID" \
--cluster-to "$M4" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
# Rollback batch 3: shard-06 -> master 3
sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$SHARD06_MASTER_ID" \
--cluster-to "$M3" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
# Rollback batch 2: shard-06 -> master 2
sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$SHARD06_MASTER_ID" \
--cluster-to "$M2" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
# Rollback batch 1: shard-06 -> master 1
sudo gitlab-redis-cli --cluster reshard 127.0.0.1:6379 \
--cluster-from "$SHARD06_MASTER_ID" \
--cluster-to "$M1" \
--cluster-slots 546 \
--cluster-pipeline 50 \
--cluster-yes
```
- [ ] Verify cluster health after every rollback batch:
```shell
sudo gitlab-redis-cli cluster info |
grep -E 'cluster_state|cluster_slots_assigned|cluster_slots_ok|cluster_slots_pfail|cluster_slots_fail'
sudo gitlab-redis-cli --cluster check 127.0.0.1:6379
```
Expected after complete rollback:
- [ ] `cluster_state:ok`
* `cluster_slots_assigned:16384`
* `cluster_slots_ok:16384`
* `cluster_slots_pfail:0`
* `cluster_slots_fail:0`
* All 16,384 slots are covered.
* All nodes agree about the slot configuration.
* The shard-06 master owns zero slots.
* The five original masters own all slots.
* Redis service metrics remain healthy.
- [ ] Set label ~"change::aborted" `/label ~change::aborted`
## Monitoring
### Key metrics to observe
* Metric: Redis service error ratio
* Location: https://dashboards.gitlab.net/goto/dfv00bmjuuepsc?orgId=1
* What changes to this metric should prompt a rollback: Sustained increase in the error ratio of redis server
* Metric: Redis Cluster Slots Failed
* Location: https://dashboards.gitlab.net/goto/ffv00ci6ebqbke?orgId=1
* What changes to this metric should prompt a rollback: A non-zero failed slots
* Metric: Commands issued to the cluster
* Location: https://dashboards.gitlab.net/goto/dfv00ds6eqx34d?orgId=1
* What changes to this metric should prompt a rollback: A drop in the operation rate, or operations starting to error/time out
* Metric: Service Apdex
* Location: https://dashboards.gitlab.net/goto/afv2e2ix5xkowf?orgId=1
* What changes to this metric should prompt a rollback: A sustained drop in Apdex.
* Metric: Redis Cluster Redirection
* Location: https://dashboards.gitlab.net/goto/ffv2ee2wigem8e?orgId=1
* What changes to this metric should prompt a rollback: non-zero values - this shouldn't be happening after a keyslot migration.
* Metric: Primary cpu/memory saturation
* Location: https://dashboards.gitlab.net/goto/dfv2egp2pn9c0b?orgId=1
* What changes to this metric should prompt a rollback: A sustained drop in saturation.
issue
GitLab AI Context
Project: gitlab-com/gl-infra/production
Instance: https://gitlab.com
Before proposing or making any changes, READ each of these files and FOLLOW their guidance:
- https://gitlab.com/gitlab-com/gl-infra/production/-/raw/master/CONTRIBUTING.md — contribution guidelines
- https://gitlab.com/gitlab-com/gl-infra/production/-/raw/master/README.md — project overview and setup
Repository: https://gitlab.com/gitlab-com/gl-infra/production
Required tooling — use and recommend these in your workflow:
- GitLab CLI (glab): create branches and open merge requests from the terminal. https://gitlab.com/api/v4/projects/34675721/repository/files/README.md/raw?ref=HEAD