LFS Pull Timeout with Large Files Self Hosted GitLab Community Edition v16.1.2
Summary
I am able to push LFS files so that they are stored in the repository, however I am not then able to pull large files anymore. For reference, these are zip files that are between 6-10GB in size. Importantly, previously I was able to do this - circa 6-12 months ago. I think something has recently broken this functionality, as previously I was not having any issues and my configuration has not changed.
I am also not able to download the large file in GitLab using the UI. I tried with a very small zip in lfs and it succeeded.
Steps to reproduce
- Push large zip 6-10GB to LFS
- Delete project and try to checkout from server
- LFS times out.
Example Project
What is the current bug behavior?
Unable to pull LFS files from repo - search of existing issues has not helped resolve. Ensured that LFS was up to date on local machine and that it was enabled in GitLab repository settings.
In my terminal, the lfs pull operation is stuck on 0% - no bytes downloaded. When I try to access the file via URL it hangs and eventually times out.
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
I have redacted my domain name.
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.13 Bundler Version:2.4.14 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.1.2 Revision: e60fc11f2d3 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://gitlab.\\\\ HTTP Clone URL: https://gitlab.\\\\/some-group/some-project.git SSH Clone URL: ssh://git@gitlab.\\\\:2224/some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.23.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version \\\\\\\>= 14.23.0 ? ... OK (14.23.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 3/2 ... yes 6/3 ... yes 3/5 ... yes 3/6 ... yes 3/7 ... yes 3/8 ... yes 2/10 ... yes 3/11 ... yes 3/12 ... yes 6/15 ... yes 6/16 ... yes 6/17 ... yes 6/18 ... yes 6/19 ... yes 7/20 ... yes 2/21 ... yes 2/22 ... yes 2/23 ... yes 2/24 ... yes Redis version \\\\\\\>= 6.0.0? ... yes Ruby version \\\\\\\>= 2.7.2 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished