2023-07-05: [GPRD][PG14][Registry] Standbys and backups corrupted
Customer Impact
This is a backstage issue that is not user impacting at the moment. We are working to restore secondary nodes.
Current Status
During investigation on https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24051 we found that both standbys of the registry cluster have data corruption. This also means that all recent backups are corrupted as well.
We successfully recreated all nodes, 1-3. The cluster is now in the desired state.
root@patroni-registry-v14-02-db-gprd.c.gitlab-production.internal:~# gitlab-patronictl list
+--------------------------------------------------------------+---------------+---------+---------+----+-----------+---------------------+
| Member | Host | Role | State | TL | Lag in MB | Tags |
+ Cluster: gprd-pg14-patroni-registry (7243063938146092911) ---+---------------+---------+---------+----+-----------+---------------------+
| patroni-registry-v14-01-db-gprd.c.gitlab-production.internal | 10.220.27.101 | Leader | running | 2 | | |
+--------------------------------------------------------------+---------------+---------+---------+----+-----------+---------------------+
| patroni-registry-v14-02-db-gprd.c.gitlab-production.internal | 10.220.27.102 | Replica | running | 2 | 0 | |
+--------------------------------------------------------------+---------------+---------+---------+----+-----------+---------------------+
| patroni-registry-v14-03-db-gprd.c.gitlab-production.internal | 10.220.27.103 | Replica | running | 2 | 0 | nofailover: true |
| | | | | | | noloadbalance: true |
+--------------------------------------------------------------+---------------+---------+---------+----+-----------+---------------------+
There is one artefact, node 05
, which needs to be removed.
📝 Summary for CMOC notice / Exec summary:
- Customer Impact: No customer impact.
- Service Impact: ServiceContainer Registry
- Impact Duration: 17:18 UTC declare time - end time UTC ( duration in minutes )
- Root cause: Related to https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24051 We introduced corruption when creating the standbys with rsync.
📚 References and helpful links
Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.