Skip to content

Perfectly normal repository causes gitaly to hang

Summary

While generating the pages list for a moderately-sized Wiki, gitaly starts gobbling up gigabytes of memory and is eventually killed. Likely relatedly, pushing the repository to gitlab.com hung.

Steps to reproduce

I have found two manifestations of the issue:

  • Create a new project. Push this repository to the Wiki and try viewing the Wiki pages list. The request will time out; looking at the server logs you will likely see that the gitaly process was killed by the OOM killer.

  • Try pushing the same repository to gitlab.com. This will fail with,

    $ git push git@gitlab.com:bgamari/ghc-wiki master
    Enumerating objects: 81883, done.
    Counting objects: 100% (81883/81883), done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (21457/21457), done.
    Writing objects: 100% (81883/81883), 18.21 MiB | 3.22 MiB/s, done.
    Total 81883 (delta 60224), reused 81715 (delta 60134)
    remote: Resolving deltas: 100% (60224/60224), done.
    remote: GitLab: Push operation timed out
    remote: 
    remote: Timing information for debugging purposes:
    remote: Running checks for ref: master
    remote: Checking if you are allowed to push... (13.8ms)
    remote: Checking if default branch is being deleted... (0.38ms)
    remote: Checking if you are force pushing to a protected branch... (2.05ms)
    remote: Checking if you are allowed to push to the protected branch... (6.32ms)
    remote: Scanning repository for blobs stored in LFS and verifying their files have been uploaded to GitLab... (23.68ms)
    remote: Checking if branch follows the naming patterns defined by the project... (8.0ms)
    remote: Validating diff contents... (cancelled after 46800.76ms)
    To gitlab.com:bgamari/ghc-wiki
     ! [remote rejected]     master -> master (pre-receive hook declined)
    error: failed to push some refs to 'git@gitlab.com:bgamari/ghc-wiki'

Example Project

https://github.com/bgamari/ghc-wiki-import

What is the current bug behavior?

Gitaly blows up.

What is the expected correct behavior?

Gitaly shouldn't blow up.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

This happens on gitlab.com and our local 11.7 installation.

Possible fixes

No idea.

Edited by Ben Gamari