DeadlineExceeded - waiting for packfile negotiation: context canceled for method SSHUploadPackWithSidechannel when fetching repo on slow connections

Support Request for the Gitaly Team

The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.

This request template is part of Gitaly Team's intake process.

Customer Information

Salesforce Link:

Zendesk Ticket: https://gitlab.zendesk.com/agent/tickets/414684

Installation Size:

Architecture Information: Omnibus

Slack Channel:

Additional Information:

Support Request

Severity

severity3

Problem Description

This is a continuation of a Slack Thread

Self-Managed customer is reporting that users on VPN/low bandwidth connection are seeing waiting for packfile negotiation errors after 10 mins when fetching a repo with git fetch --unshallow using SSH:

fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

The Gitlay logs (accessible on the ticket) show:

"command.real_time_ms": 606551,
"command.spawn_token_wait_ms": 0,
"command.system_time_ms": 39,
"command.user_time_ms": 55,
"correlation_id": "01H28769XSQNB9D792PERH9KAY",
"error": "waiting for packfile negotiation: context canceled",
"grpc.code": "DeadlineExceeded",
"grpc.meta.client_name": "gitlab-shell-git-upload-pack",
"grpc.meta.deadline_type": "none",
"grpc.meta.method_type": "unary",
"grpc.method": "SSHUploadPackWithSidechannel",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPackWithSidechannel",

The error is raised in the code here, using a timeout created here which has a hardcoded value of 10 mins set here.

This is not configurable so there is no option to increase the setting to allow for these low bandwidth connections.

I have passed the context on why the timeout exists:

in order to avoid a denial-of-service scenario where an adversary performs many pushes, but simply doesn't write any data. The result would be that we end up consuming all file descriptors.

The customer believes there is a defect and the timeout should not apply to fetches.

Troubleshooting Performed

What specifically do you need from the Gitaly team

1. The operation is a git fetch --unshallow command not a git push (this is in relation to the DOS explanation) and bug is that the unshallow fetch is defected by the timeout, is this assumption correct?

   • I did respond that the timeout is applicable to any code path where the sshUploadPack function is called on the Gitaly Server and one of those paths is invoked with git fetch --unshallow which involves packfile negotiation, is there anything you can add/correct here?

2. Why does the timeout apply to SSH but not HTTPS?

3. Should this be considered a Defect or Feature Request going forward?

Author Checklist

• Customer information provided
• Severity realistically set
• Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team

/cc @mjwood @andrashorvath @jcaigitlab @john.mcdonnell