500 errors caused by corrupt packed-refs (fatal: unterminated line in /var/opt/gitlab/[.....]/aa/bb/aabb*.git/packed-refs: 55b15c21\n)

Support Request for the Gitaly Team

The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.

This request template is part of Gitaly Team's intake process.

Customer Information

Salesforce Link: https://gitlab.my.salesforce.com/00161000006fjFiAAI

Zendesk Ticket: https://gitlab.zendesk.com/agent/tickets/353124

Installation Size: large; including three Praefect and three Gitaly nodes.

Architecture Information:

Slack Channel:

Additional Information:

Support Request

Severity

severity3

Problem Description

Initially one project was inaccessible owing to

fatal: unterminated line in [path-to-git-repo]/.git/packed-refs: 55b15c21\n

errors on one gitaly node in the cluster. I assume this is the primary.

The file is truncated part way through the SHA for a tag ref., as in

288336f9040fa1f641797d8b1e2d155a0237def7	refs/tags/v9.5.6
^65f8414c691201fe1de89f16cc9929b20f866702
0f424e022f13fd6140bd8d67b53260091b79df96	refs/tags/v9.5.8
^fbffc274b5df7e79222aeaf55bf5b3b9a466f4cc
720829168bddc118ccfec6e759914f35ab0bf4bb	refs/tags/v9.5.9
^32be8c907edefcaa1b2db7aa5d08245f1810d707
55b15c21

Troubleshooting Performed

We have logs; I've requested git ls-remote output, praefect metadata output and rake gitlab:praefect:replicas output.

What specifically do you need from the Gitaly team

Is my action plan OK; should I do anything else, have I misunderstood anything?

Action plan:

• test this plan
• set the repo read only using the rails console - https://docs.gitlab.com/ee/administration/raketasks/storage.html#read-only-projects
• verify the status of the three copies of the repository
• use accept dataloss to select a different copy and over-write the two other copies.

If git ls-remote and rake gitlab:praefect:replicas run against the three copies suggests that there is no quorum amongst the uncorrupted copies, then analysis will be needed to pick which copy to keep.

I would also consider manually fixing packed-refs once the repo is read only to get an extra opinion on the refs the three repos can agree on.

Author Checklist

• Customer information provided
• Severity realistically set
• Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team

/cc @mjwood @andrashorvath @john.mcdonnell