Skip to content

LFS Access forbidden error after updating from 14.10 to 15.0

Summary

The LFS object storage stopped working our self-hosted (omnibus install) GitLab server after updating from 14.10.2 to 15.0.2. And the problems have persisted after updating to 15.0.3.

Our LFS object storage is set up to use an AWS S3 bucket. After the upgrade, we are now getting "Access forbidden" errors when trying to push LFS objects to the server. It happens both on existing projects with LFS objects and on brand new projects. We can pull existing LFS objects down just fine. We just can't upload.

A git push results in:

Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
LFS: Access forbidden. Check your access level.
error: failed to push some refs to 'gitlab.<ourdomain>.com:<user>/lfstest.git'

This happens for several (all) users and happens with SSH connections or with project access tokens (with the write_repository scope and a Maintainer role). It happens with Git 2.30.2 and git-lfs 2.13.2 (on Windows). And it happens even after updating the client Git and Git-LFS installs to the latest released versions (git 2.36.1 and git-lfs 3.1.4).

We have verified that the AWS EC2 instance running GitLab has a role with a policy that allows the necessary Get/Put actions on the S3 bucket. The exact same bucket, role, and policies were in place when this worked in Version 14.10.

I don't know if it's at all relevant, but we also updated PostgreSQL to version 13 before updating to 15.0.2.

Steps to reproduce

  • Start with a Ubuntu server running GitLab 14.10.2, omnibus install, role/policies for <bucket_name> S3 bucket. Relevant part of the config in /etc/gitlab/gitlab.rb: 
    gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "<bucket_name>/gitlab-lfs-objects"
gitlab_rails['lfs_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => '<region>',
  'use_iam_profile' => true
}
  • Pushing LFS objects to a repo works
  • Run gitlab-ctl pg-upgrade -V 13 to update PostgreSQL
  • Update to GitLab 15.0.2 (apt update && apt install gitlab-ee=15.0.2)
  • Notice the problem; Pushing LFS object to a repo fails with: LFS: Access forbidden. Check your access level.
  • Update to GitLab 15.0.3 (apt update && apt install gitlab-ee=15.0.3)
  • Problem persists
  • Change the /etc/gitlab/gitlab.rb config file to use the consolidated object storage approach: 
    gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false # or true, error persists either way
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'region' => '<region>',
  'use_iam_profile' => true
}
gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<bucket_name>/gitlab-lfs-objects"
gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['uploads']['enabled'] = false
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false
  • Problem persists

What is the current bug behavior?

Pushing LFS objects results in git-lfs error message:

LFS: Access forbidden. Check your access level.

What is the expected correct behavior?

LFS objects are uploaded and stored in the configured AWS S3 bucket.

Relevant logs and/or screenshots

The relevant lines from production.log seem to be:

Started PUT "/<user>/lfstest.git/gitlab-lfs/objects/b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531/6194/authorize" for <IP address> at 2022-06-17 15:36:45 +0000
Processing by Repositories::LfsStorageController#upload_authorize as HTML
  Parameters: {"repository_path"=>"<user>/lfstest.git", "oid"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "size"=>"6194"}
Completed 200 OK in 418ms (Views: 1.0ms | ActiveRecord: 5.0ms | Elasticsearch: 0.0ms | Allocations: 47529)
Started PUT "/<user>/lfstest.git/gitlab-lfs/objects/b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531/6194" for <IP address> at 2022-06-17 15:36:45 +0000
Processing by Repositories::LfsStorageController#upload_finalize as HTML
  Parameters: {"file.gitlab-workhorse-upload"=>"<...>", "file.md5"=>"<...>", "file.name"=>"", "file.path"=>"", "file.remote_id"=>"1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0", "file.remote_url"=>"https://s3.<region>.amazonaws.com/<bucket_name>/gitlab-lfs-objects/tmp/uploads/1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0?X-Amz-Expires=15300&X-Amz-Date=20220617T153645Z&X-Amz-Security-Token=<...>&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<...>&X-Amz-SignedHeaders=host&X-Amz-Signature=<...>", "file.sha1"=>"<...>", "file.sha256"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "file.sha512"=>"<...>", "file.size"=>"6194", "file.upload_duration"=>"0.018153652", "file"=>#<UploadedFile:0x00007f9318c994d0 @size=6194, @upload_duration=0.018153652, @content_type="application/octet-stream", @original_filename="unnamed", @sha256="b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", @remote_id="1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0">, "repository_path"=>"<user>/lfstest.git", "oid"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "size"=>"6194"}
Completed 403 Forbidden in 108ms (Views: 0.3ms | ActiveRecord: 6.3ms | Elasticsearch: 0.0ms | Allocations: 19319)

Similar lines are repeated several times as the git-lfs client retries.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 20.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.3.15
Rake Version:   13.0.6
Redis Version:  6.2.6
Sidekiq Version:6.4.0
Go Version:     unknown

GitLab information
Version:        15.0.3-ee
Revision:       b7e551ef451
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.6
URL:            http://gitlab.ourdomain.com
HTTP Clone URL: http://gitlab.ourdomain.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.ourdomain.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.3.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Edited by Murray Johnson