FindLocalBranches Errors
Support Request for the Gitaly Team
Customer Information
Architecture Information:
Docker Container Hosting approx 7k Users
Support Request
Severity
Projects cant be moved so kind of medium to low?
Problem Description
FindLocalBranches errors and I cant move projects in the Gitlab UI. We also saw that forks from that repo exist.
Gitlab 14.9.2
Troubleshooting Performed
went through all logs of gitlabweb -> praefect -> gitaly
found:
{
"correlation_id": "01G1TMZX594VF40X9ZJ0R970CP",
"error": "accessor call: route repository accessor: consistent storages: repository \"default\"/\"non-hashed-path.git\" not found",
"grpc.code": "NotFound",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.meta.method_type": "bidi_stream",
"grpc.method": "FindLocalBranches",
"grpc.request.deadline": "2022-04-29T12:33:46.010",
"grpc.request.fullMethod": "/gitaly.RefService/FindLocalBranches",
"grpc.service": "gitaly.RefService",
"grpc.start_time": "2022-04-29T12:33:36.011",
"grpc.time_ms": 1.206,
"level": "info",
"msg": "finished streaming call with code NotFound",
"peer.address": "REDACTED:55054",
"pid": 622,
"relative_path": "non-hashed-path.git",
"remote_ip": "REDACTED",
"span.kind": "server",
"system": "grpc",
"time": "2022-04-29T12:33:36.012Z",
"username": "HenrySachs",
"virtual_storage": "default"
}
The other Components voted for the transaction with an OK.
checked the hashes on all 3 gitaly nodes
root@d846e1484c0b:/# gitlab-rake "gitlab:praefect:replicas[XXX]"
Project name | gitaly-3 (primary) | gitaly-1 | gitaly-2
---------------------------------------------------------------------------------------------------------------------------------------------
XXX | 3d2c592e4e1490166bb27493e2cc0d12bd0666ce | 3d2c592e4e1490166bb27493e2cc0d12bd0666ce | 3d2c592e4e1490166bb27493e2cc0d12bd0666ce
run git fsck
on all repos.
What specifically do you need from the Gitaly team
How can praefect send a not found for the branches in this case?
The Storage is consistent in all 3 of our Gitaly nodes. What can I do to make this repo "moveable" again?
What could I do better to troubleshoot the error?
Author Checklist
-
Customer information provided -
Severity realistically set -
Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team
/cc @mjwood