LFS Error 422, Gitlab 11.3.0

Okay I've already gone over everything I can find both on here and through google. I'm sure it's not an issue with GITLFS version (2.4.2 or 2.5.0 or 2.5.1 or 2.5.2). I've tried them all. (Infact, I was using 2.4.1 in the first place..)

I suspect that it must be something with my configuration.

I have tried without passenger, going back to using Unicorn, but still keeping non-bundled nginx-- same problem. I'm also not attempting to use HTTPS or anything.

I always get 422 Unprocessable Entity

22:00:51.395635 trace git-lfs: HTTP: PUT http://gitlab.saibertek.systems/fixgames/tivr_ue.git/gitlab-lfs/objects/b58749df7171b26777deead0b00bb0dd695aa8f143820abcc09f62f16ae56105/5478537
> PUT /fixgames/tivr_ue.git/gitlab-lfs/objects/b58749df7171b26777deead0b00bb0dd695aa8f143820abcc09f62f16ae56105/5478537 HTTP/1.1
> Host: gitlab.saibertek.systems
> Authorization: Basic * * * * *
> Content-Length: 5478537
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.5.2 (GitHub; windows amd64; go 1.10.3; git 8e3c5c93)
>
22:00:52.607753 trace git-lfs: HTTP: 422B | 6.4 MB/s

< HTTP/1.1 422 Unprocessable Entity
< Transfer-Encoding: chunked
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 29 Sep 2018 13:00:52 GMT
< Server: nginx + Phusion Passenger 5.3.5
< Status: 422 Unprocessable Entity
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Powered-By: Phusion Passenger 5.3.5
< X-Request-Id: 670d48ed-95fd-4746-ad25-b6aef7e217c5
< X-Runtime: 0.012887
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block

My gitlab.rb:

nginx['enable'] = false
unicorn['enable'] = false

external_url 'http://gitlab.saibertek.systems'
web_server['external_users'] = ['www-data']

gitlab_rails['internal_api_url'] = 'http://gitlab.saibertek.systems'
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/lfs-objects"
gitlab_rails['lfs_object_store_enabled'] = false
gitlab_rails['lfs_object_store_background_upload'] = true

I also have an alternative location for the lfs-objects, which I changed back to default to see if it would fix the issue, but it did not.

my nginx vhost configuration which is pretty much verbatim from the gitlab documentation.. I just increased the client_max_body_size in the server block, and of course set my domain names.

    upstream gitlab-workhorse {
      server unix://var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;
    }
    
    server {
        listen *:80;
        server_name gitlab.saibertek.systems;
        server_tokens off;
        root /opt/gitlab/embedded/service/gitlab-rails/public;
        
        client_max_body_size 4096m;
        
        access_log  /var/log/nginx/gitlab_access.log;
        error_log   /var/log/nginx/gitlab_error.log;
        
        # Ensure Passenger uses the bundled Ruby version
        passenger_ruby /opt/gitlab/embedded/bin/ruby;
        
        # Correct the $PATH variable to included packaged executables
        passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";
        
        # Make sure Passenger runs as the correct user and group to
        # prevent permission issues
        passenger_user git;
        passenger_group git;
        
        # Enable Passenger & keep at least one instance running at all times
        passenger_enabled on;
        passenger_min_instances 1;
        
        location ~ ^/[\w\.-]+/[\w\.-]+/(info/refs|git-upload-pack|git-receive-pack)$ {
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        location ~ ^/[\w\.-]+/[\w\.-]+/repository/archive {
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        location ~ ^/api/v3/projects/.*/repository/archive {
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        # Build artifacts should be submitted to this location
        location ~ ^/[\w\.-]+/[\w\.-]+/builds/download {
            client_max_body_size 0;
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        # Build artifacts should be submitted to this location
        location ~ /ci/api/v1/builds/[0-9]+/artifacts {
            client_max_body_size 0;
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        # Build artifacts should be submitted to this location
        location ~ /api/v4/jobs/[0-9]+/artifacts {
            client_max_body_size 0;
            # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
            error_page 418 = @gitlab-workhorse;
            return 418;
        }
        
        
        # For protocol upgrades from HTTP/1.0 to HTTP/1.1 we need to provide Host header if its missing
        if ($http_host = "") {
            # use one of values defined in server_name
            set $http_host_with_default "gitlab.saibertek.systems";
        }
        
        if ($http_host != "") {
            set $http_host_with_default $http_host;
        }
        
        location @gitlab-workhorse {
            ## https://github.com/gitlabhq/gitlabhq/issues/694
            ## Some requests take more than 30 seconds.
            proxy_read_timeout      3600;
            proxy_connect_timeout   300;
            proxy_redirect          off;
            
            # Do not buffer Git HTTP responses
            proxy_buffering off;
            
            proxy_set_header    Host                $http_host_with_default;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Proto   $scheme;
            
            proxy_pass http://gitlab-workhorse;
            
            ## Pass chunked request bodies to gitlab-workhorse as-is
            proxy_request_buffering off;
            proxy_http_version 1.1;
        }
        
        ## Enable gzip compression as per rails guide:
        ## http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression
        ## WARNING: If you are using relative urls remove the block below
        ## See config/application.rb under "Relative url support" for the list of
        ## other files that need to be changed for relative url support
        location ~ ^/(assets)/ {
            root /opt/gitlab/embedded/service/gitlab-rails/public;
            gzip_static on; # to serve pre-gzipped version
            expires max;
            add_header Cache-Control public;
        }
        
        error_page 502 /502.html;
    }

If there is any more information I need to provide to narrow down if infact this really is a bug, or just a configuration problem, Please let me know!

Edited by 🤖 GitLab Bot 🤖