Http 413 when adding lfs objects over 100m
Summary
I am running the docker gitlab-ce latest image version says 11.9.8
When pushing lfs objects over 100m I am getting a http 413 error. The same file pushes successfully if not using lfs.
I have set nginx['client_max_body_size'] to different value from 0 to 10000 and have not seen any changes.
I have also set the "Maximum artifacts size (MB)" "Maximum push size (MB)" and "Maximum attachment size (MB)" to match when I update the client_max_body_size
I have my reverse proxy turned off and am pointing directly at gitlab to minimize points of failure.
Steps to reproduce
Create a new project
Commit and push .attributes file with
*.zip filter=lfs diff=lfs merge=lfs -text
Commit and push files ranging in size from 1 to 300 mb
What is the current bug behavior?
Any file in lfs over 100mb is not pushed with a http 413 error.
What is the expected correct behavior?
files should be allowed up to the size defined in nginx['client_max_body_size']
Relevant logs
Relevant logs
production.logStarted GET "/s3rraph/test.git/info/refs?service=git-receive-pack" for 162.158.106.15 at 2019-04-16 11:14:33 +0000 Processing by Projects::GitHttpController#info_refs as / Parameters: {"service"=>"git-receive-pack", "namespace_id"=>"s3rraph", "project_id"=>"test.git"} Filter chain halted as :authenticate_user rendered or redirected Completed 401 Unauthorized in 19ms (Views: 0.9ms | ActiveRecord: 2.7ms) Started GET "/s3rraph/test.git/info/refs?service=git-receive-pack" for 162.158.106.15 at 2019-04-16 11:14:34 +0000 Processing by Projects::GitHttpController#info_refs as / Parameters: {"service"=>"git-receive-pack", "namespace_id"=>"s3rraph", "project_id"=>"test.git"} Completed 200 OK in 183ms (Views: 0.6ms | ActiveRecord: 6.5ms) Started GET "/help" for 127.0.0.1 at 2019-04-16 11:14:34 +0000 Processing by HelpController#index as / Completed 200 OK in 709ms (Views: 698.8ms | ActiveRecord: 1.0ms) Started GET "/s3rraph/test.git/info/refs?service=git-upload-pack" for 108.162.246.20 at 2019-04-16 11:14:35 +0000 Processing by Projects::GitHttpController#info_refs as / Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"s3rraph", "project_id"=>"test.git"} Filter chain halted as :authenticate_user rendered or redirected Completed 401 Unauthorized in 23ms (Views: 1.1ms | ActiveRecord: 2.6ms) Started GET "/s3rraph/test.git/info/refs?service=git-upload-pack" for 108.162.246.20 at 2019-04-16 11:14:35 +0000 Processing by Projects::GitHttpController#info_refs as / Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"s3rraph", "project_id"=>"test.git"} Completed 200 OK in 187ms (Views: 0.8ms | ActiveRecord: 6.7ms) Started POST "/s3rraph/test.git/info/lfs/locks/verify" for 71.9.71.169 at 2019-04-16 11:14:36 +0000 Processing by Projects::LfsLocksApiController#verify as JSON Parameters: {"ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_locks_api"=>{"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 181ms (Views: 0.5ms | ActiveRecord: 9.1ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:37 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 786ms (Views: 1.3ms | ActiveRecord: 89.1ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:38 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 210ms (Views: 0.8ms | ActiveRecord: 10.6ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:39 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 179ms (Views: 0.6ms | ActiveRecord: 9.1ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:40 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 190ms (Views: 0.6ms | ActiveRecord: 9.5ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:40 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 195ms (Views: 3.6ms | ActiveRecord: 9.0ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:41 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 203ms (Views: 0.8ms | ActiveRecord: 9.7ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:42 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 218ms (Views: 1.5ms | ActiveRecord: 9.7ms) Started GET "/-/metrics" for 127.0.0.1 at 2019-04-16 11:14:42 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 29ms (Views: 0.6ms | ActiveRecord: 0.0ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:42 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 179ms (Views: 0.6ms | ActiveRecord: 9.2ms) Started POST "/s3rraph/test.git/info/lfs/objects/batch" for 71.9.71.169 at 2019-04-16 11:14:43 +0000 Processing by Projects::LfsApiController#batch as JSON Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"s3rraph", "project_id"=>"test.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920", "size"=>152317258}],"ref"=>{"name"=>"refs/heads/master"}}} Completed 200 OK in 176ms (Views: 0.4ms | ActiveRecord: 9.4ms) Started GET "/s3rraph/test.git/info/refs?service=git-upload-pack" for 108.162.246.80 a
git output
$ git-lfs push origin master Locking support detected on remote "origin". Consider enabling it with: $ git config lfs.https://####.com/####/test.git/info/lfs.locksverify true LFS: Client error: https://####.com/####/test.git/gitlab-lfs/objects/348f820270c6f50f4b83765adec35a160163a2aa7680828055417c5a36322920/152317258 from HTTP 413 Uploading LFS objects: 0% (0/1), 20 KB | 6.1 KB/s, done
Details of package version
Provide the package version installation details
gitlab-ce 11.9.8-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
Environment details
- Operating System:
Docker centos atomic host
- Installation Target, remove incorrect values:
- Other:
DOCKER
- Other:
- Installation Type, remove incorrect values:
- New Installation
- Upgrade from version
11.8.0
- Is there any other software running on the machine: stopped all other containers
- Is this a single or multiple node installation? single
- Resources
- CPU:
Dual xeon E5620
- Memory total:
16gb
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://######.com' gitlab_rails['lfs_enabled'] = true nginx['client_max_body_size'] = '1024m' nginx['listen_addresses'] = ['*']