Skip to content

Forking projects with large number of LFS files fail with no error message

Summary

Forking a project via the web interface with more that 100,000 LFS files fails to successfully fork. Project appears to successfully fork as in no error messages are returned and a new project is created within GitLab. On trying to clone new project you get missing object errors

$ git-lfs.exe filter-process Error downloading object: Images/C/starmap.data (2036359): Smudge error: Error downloading Images/C/starmap.data (20363595f309da9626111937d6d75555530937914527abbd6d81132764434aa7): [20363595f309da9626111937d6d75555530937914527abbd6d81132764434aa7] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it

Steps to reproduce

  1. Create a new project with a Readme
  2. Clone project
  3. in cloned project install LFS, track a file type and add 100,000+ small LFS files
  4. commit and push repo to GitLab
  5. In Gitlab fork the project

What is the current bug behavior?

Fork returns successful and new project appears in Gitlab. There is no warning/error messages that anything went wrong.

Try cloning project and cloning fails with "[404] Object does not exist on the server or you don't have permissions to access it" error messages

Forking process should return a Error or Warning that fork failed.

Relevant logs and/or screenshots

Gitlab records the fact that the process failed but is silent about it. If you get new forked project details using API "GET /projects/:id". There is status details.

'import_status' => 'failed',
'import_error' => 'Every fork attempt has failed: Unable to fork project 17400 for repository @hashed/a9/1d/a91d4278281f4f5e96c5d370911d65dc95d56bb00217f97051638928cdcf1261 -> @hashed/70/63/706339408a73ca16a474523ac2e14bde9c0a12336198d7636bb508406acdc853: Failed to create fork repository. Please try again.',

Also in /var/log/gitlab/sidekiq/current

"exception.class":"RuntimeError","exception.message":"Unable to fork project 17400 for repository @hashed/a9
/1d/a91d4278281f4f5e96c5d370911d65dc95d56bb00217f97051638928cdcf1261 -\u003e 
@hashed/70/63/706339408a73ca16a474523ac2e14bde9c0a12336198d7636bb508406acdc853: Source project has too many LFS objects",

Output of checks

Expand for output related to GitLab environment info

System information
System:         Ubuntu 22.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.2.33
Bundler Version:2.3.15
Rake Version:   13.0.6
Redis Version:  6.2.11
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        15.11.2-ee
Revision:       916d24d1e48
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.8
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

Results of GitLab application Check

Possible fixes

Not a fix but has a arbitrary size set is is why it fails in the first place

/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/lfs_pointers/lfs_link_service.rb

      4 module Projects
      5   module LfsPointers
      6     class LfsLinkService < BaseService
      7       TooManyOidsError = Class.new(StandardError)
      8 
      9       MAX_OIDS = 100_000
     10       BATCH_SIZE = 1000