Unable to view the commit history of files due to FindCommits GRPC call timing out (w/ reproducer)
Support Request for the Gitaly Team
Please help investigate; steps to reproduce are at https://gitlab.com/gitlab-com/dev-sub-department/section-dev-request-for-help/-/issues/111#note_1446489018 (this is a proxy issue for scheduling reasons)
The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.
This request template is part of Gitaly Team's intake process.
Customer Information
Salesforce Link: https://gitlab.my.salesforce.com/00161000004zq6rAAA
Zendesk Ticket: https://gitlab.zendesk.com/agent/tickets/413769
Installation Size: 2250 seats
Architecture Information: See https://gitlab.com/gitlab-com/dev-sub-department/section-dev-request-for-help/-/issues/111
Slack Channel: https://gitlab.slack.com/archives/C05CFGQ6LKE
Additional Information:
Support Request
Determine why the FindCommits GRPC call is taking so long to return the commit history, and provide a fix
Severity
Problem Description
This customer is unable to view the commit history of a file within a large project (46 GB).
The FindCommits GRPC method in Gitaly seems to be timing out just before the 30 second timeout.
After working with the Gitaly team, the FindCommits GRPC method calls 2 underlying git commands:
git loggit cat-file
If we run these git commands directly on the bare repository or a local clone, they complete within seconds.
When Gitaly calls git cat-file, it does this via the --batch-check / --batch arguments, so input is passed in via stdin (this differs from how we call git cat-file above because I wasn't able to simulate this part).
It's interesting that when this process is spawned, it consumes 100% CPU on 1 core until either the Gitaly timeout is hit or it successfully completes. However, this process takes substantially longer than running the git commands above directly.
Why is this the case?
Troubleshooting Performed
We were unable to examine a copy of the customer's repository for reproduction purposes.
I was able to reproduce this on the Linux project however - see https://gitlab.com/gitlab-com/dev-sub-department/section-dev-request-for-help/-/issues/111#note_1446489018 for the steps.
What specifically do you need from the Gitaly team
Author Checklist
-
Customer information provided -
Severity realistically set -
Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team