Mysteriously Missing git repository data after upgrading to 16.
Support Request for the Gitaly Team
The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.
This request template is part of Gitaly Team's intake process.
Customer Information
Salesforce Link:
Zendesk Ticket: https://gitlab.zendesk.com/agent/tickets/490385
Installation Size: 3k CNG
Architecture Information:
CNG - AWS Gitaly Cluster
3 Praefect nodes 4 Gitaly nodes
Slack Channel:
Additional Information:
Support Request
Severity
Problem Description
Customer has reported that after upgrading their dev instance multiple repositories are now showing as empty that previously had data. Their upgrade process does include recreating AMI instances and re-attaching but the level of data loss seems very high.
I could use some help figuring out what to look for to get to the bottom of this. I have seen nothing in the attached logs that would give signs of what happened.
Troubleshooting Performed
checked data-loss command and found some repos behind but no total "missing" checked Gitaly and Praefect logs to see if get sense of repo damage checked mounts and permissions
What specifically do you need from the Gitaly team
Help identifying possible causes to investigate further.
Author Checklist
-
Customer information provided -
Severity realistically set -
Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team
/cc @mjwood @andrashorvath @jcaigitlab @john.mcdonnell @gerardo