Data loss - Specific files and commit history sporadically disappear from repository
Support Request for the Gitaly Team
The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.
This request template is part of Gitaly Team's intake process.
Customer Information
Salesforce Link: https://gitlab.my.salesforce.com/0016100001dIq8l
Zendesk Ticket:
- https://gitlab.zendesk.com/agent/tickets/506425
- https://gitlab.zendesk.com/agent/tickets/428850 (cc @rmongare)
Installation Size: 4650 seats
Architecture Information: 3k reference architecture
Support Request
Severity
Per the prioritization framework, I'd prioritize this as severity1 due to the data-loss aspect.
As the impact is limited to two files on a single repository, and the fact a workaround exists (push the file from an unaffected local clone), I'll bump this down to severity3
From the customer perspective, this is a continued annoyance that lowers the trust in GitLab as a product. They do however agree that the problem might arise due to their specific workflow on this repo.
Problem Description
The customer reports they "lose" to specific files (commitHash.yaml
, master_release.xml
) in a repository on a regular basis. We have one closed and one ongoing ticket about this issue.
The repository is being force pushed to on a regular basis, during CI/CD jobs.
Troubleshooting Performed
- Checking the Git logs for commits that'd have deleted or renamed the files
git log --all --diff-filter=DR --full-history -- {commitHash.yaml,master_release.xml}
- No such commits were found for either files.
-
git log
does not show any history for the files beside the "restore" commits.
- Checking Gitaly logs for issues
- Checking the hosts syslog / dmesg for signs of data-loss
What specifically do you need from the Gitaly team
Help us identify the root-cause for the data loss.
Author Checklist
-
Customer information provided -
Severity realistically set -
Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team
/cc @mjwood @andrashorvath @jcaigitlab @john.mcdonnell @gerardo