Forking projects with large number of LFS files fail with no error message
Summary
Forking a project via the web interface with more that 100,000 LFS files fails to successfully fork. Project appears to successfully fork as in no error messages are returned and a new project is created within GitLab. On trying to clone new project you get missing object errors
$ git-lfs.exe filter-process Error downloading object: Images/C/starmap.data (2036359): Smudge error: Error downloading Images/C/starmap.data (20363595f309da9626111937d6d75555530937914527abbd6d81132764434aa7): [20363595f309da9626111937d6d75555530937914527abbd6d81132764434aa7] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it
Steps to reproduce
- Create a new project with a Readme
- Clone project
- in cloned project install LFS, track a file type and add 100,000+ small LFS files
- commit and push repo to GitLab
- In Gitlab fork the project
What is the current bug behavior?
Fork returns successful and new project appears in Gitlab. There is no warning/error messages that anything went wrong.
Try cloning project and cloning fails with "[404] Object does not exist on the server or you don't have permissions to access it" error messages
Forking process should return a Error or Warning that fork failed.
Relevant logs and/or screenshots
Gitlab records the fact that the process failed but is silent about it. If you get new forked project details using API "GET /projects/:id". There is status details.
'import_status' => 'failed',
'import_error' => 'Every fork attempt has failed: Unable to fork project 17400 for repository @hashed/a9/1d/a91d4278281f4f5e96c5d370911d65dc95d56bb00217f97051638928cdcf1261 -> @hashed/70/63/706339408a73ca16a474523ac2e14bde9c0a12336198d7636bb508406acdc853: Failed to create fork repository. Please try again.',
Also in /var/log/gitlab/sidekiq/current
"exception.class":"RuntimeError","exception.message":"Unable to fork project 17400 for repository @hashed/a9
/1d/a91d4278281f4f5e96c5d370911d65dc95d56bb00217f97051638928cdcf1261 -\u003e
@hashed/70/63/706339408a73ca16a474523ac2e14bde9c0a12336198d7636bb508406acdc853: Source project has too many LFS objects",
Output of checks
Expand for output related to GitLab environment info
System information System: Ubuntu 22.04 Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.2.33 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.11.2-ee Revision: 916d24d1e48 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
Results of GitLab application Check
Possible fixes
Not a fix but has a arbitrary size set is is why it fails in the first place
/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/lfs_pointers/lfs_link_service.rb
4 module Projects 5 module LfsPointers 6 class LfsLinkService < BaseService 7 TooManyOidsError = Class.new(StandardError) 8 9 MAX_OIDS = 100_000 10 BATCH_SIZE = 1000