"Validating diff contents" cancelled after 50 seconds when pushing new branch with tens of thousands of objects
Creating a new branch in GitLab with git push -u …GL…upstream… …local_branch…
can fail due to push rule problems (see gitlab#9326 (closed) & gitlab-foss#57067 (closed) for example), in which case the observed behavior includes:
remote: Validating diff contents... (cancelled after …ca…50k…ms…)
…
error: failed to push some refs to 'git@gitlab.com:….git'
However, it can also fail if the pushed content is somehow large (ca. 8k commits & 30k files with many renames in this case). Additional output was observed with git push -v
Enumerating objects: …ca…120k…, done.
…
remote: Checking connectivity: …ca…85k…, done.
remote: GitLab: Internal API error (502)
…
! [remote rejected] …branch…name… -> …branch…name… (pre-receive hook declined)
(although the hook declined
may be an unfitting error message in this case) and in gitlab-workhorse/current
{ …
"duration_ms": …ca…40k…,
"error": "badgateway: failed to receive response: EOF",
"level": "error",
"method": "POST",
…
"uri": "/api/v4/internal/allowed"
}
{ …
"duration_ms": …ca…40k…,
"host": "…",
"level": "info", # side note: This seems weird for a 502
"method": "POST",
…
"status": 502,
…
"uri": "/api/v4/internal/allowed",
"user_agent": "Go-http-client/1.1",
…
}
and in gitaly/current
:
{
"duration_ms": …ca…40k…,
"error": "Internal API error (502)",
"level": "error",
"method": "POST",
"msg": "Internal API error",
…
"url": "http://…/api/v4/internal/allowed"
}
The branch in question seems significantly larger than "one of the largest representative MR[s]" (internal chat) we use in staging.
Attempted solutions
- disabling all push rules (branch & file names) => not successful
- increasing Gitaly timeouts => not successful
- increasing the puma['per_worker_max_memory_mb'] setting as pioneered by gitlab#267499 (closed) => eliminated
PumaWorkerKiller: Out of memory
events that were seen before (environment: 1 Gitaly & 3 app nodes with 12 GB each), but not the here-reported issue
Successful work-around
- created a new personal project
- pushed the branch into that
- set it up as a fork of the main repo via the API.
- created an MR & merged
Attempted, more complicated work-around
-
git log --oneline local_branch..upstream_merge_target > /tmp/local_commits.txt
- the
HEAD~1234:
syntax didn't work, probably because of a too complicated branch history (seeDesired outcome
section below)
- the
- "bisecting" the branch log manually but picking a commit ID from the middle, around 25%, 10%, etc. to use in
git push upstream …local…SHA…:…upstream…branch…
- repeating 3. to push the branch up in slices/chunks, so to speak.
Not attempted
@zj-gitlab suggested to increase puma['worker_timeout'], which was overlooked before (probably by me).
Desired outcome
All this leads up to whether or how we can improve Gitaly and/our Puma performance for large/huge push events.
On the user-side, such event can most likely be avoided by pushing often. For example, when prep'ing a release branch from many topic branches => better push after each merge/rebase/cherry-pick ;-)
Reported by
A large Premium customer (internal ticket).