LFS Access forbidden error after updating from 14.10 to 15.0
Summary
The LFS object storage stopped working our self-hosted (omnibus install) GitLab server after updating from 14.10.2 to 15.0.2. And the problems have persisted after updating to 15.0.3.
Our LFS object storage is set up to use an AWS S3 bucket. After the upgrade, we are now getting "Access forbidden" errors when trying to push LFS objects to the server. It happens both on existing projects with LFS objects and on brand new projects. We can pull existing LFS objects down just fine. We just can't upload.
A git push
results in:
Uploading LFS objects: 0% (0/1), 0 B | 0 B/s, done.
LFS: Access forbidden. Check your access level.
error: failed to push some refs to 'gitlab.<ourdomain>.com:<user>/lfstest.git'
This happens for several (all) users and happens with SSH connections or with project access tokens (with the write_repository
scope and a Maintainer
role). It happens with Git 2.30.2 and git-lfs 2.13.2 (on Windows). And it happens even after updating the client Git and Git-LFS installs to the latest released versions (git 2.36.1 and git-lfs 3.1.4).
We have verified that the AWS EC2 instance running GitLab has a role with a policy that allows the necessary Get
/Put
actions on the S3 bucket. The exact same bucket, role, and policies were in place when this worked in Version 14.10.
I don't know if it's at all relevant, but we also updated PostgreSQL to version 13 before updating to 15.0.2.
Steps to reproduce
- Start with a Ubuntu server running GitLab 14.10.2, omnibus install, role/policies for
<bucket_name>
S3 bucket. Relevant part of the config in/etc/gitlab/gitlab.rb
:gitlab_rails['lfs_object_store_enabled'] = true gitlab_rails['lfs_object_store_remote_directory'] = "<bucket_name>/gitlab-lfs-objects" gitlab_rails['lfs_object_store_connection'] = { 'provider' => 'AWS', 'region' => '<region>', 'use_iam_profile' => true }
- Pushing LFS objects to a repo works
- Run
gitlab-ctl pg-upgrade -V 13
to update PostgreSQL - Update to GitLab 15.0.2 (
apt update && apt install gitlab-ee=15.0.2
) - Notice the problem; Pushing LFS object to a repo fails with:
LFS: Access forbidden. Check your access level.
- Update to GitLab 15.0.3 (
apt update && apt install gitlab-ee=15.0.3
) - Problem persists
- Change the
/etc/gitlab/gitlab.rb
config file to use the consolidated object storage approach:gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['proxy_download'] = false # or true, error persists either way gitlab_rails['object_store']['connection'] = { 'provider' => 'AWS', 'region' => '<region>', 'use_iam_profile' => true } gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<bucket_name>/gitlab-lfs-objects" gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false gitlab_rails['object_store']['objects']['uploads']['enabled'] = false gitlab_rails['object_store']['objects']['packages']['enabled'] = false gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false gitlab_rails['object_store']['objects']['pages']['enabled'] = false
- Problem persists
What is the current bug behavior?
Pushing LFS objects results in git-lfs error message:
LFS: Access forbidden. Check your access level.
What is the expected correct behavior?
LFS objects are uploaded and stored in the configured AWS S3 bucket.
Relevant logs and/or screenshots
The relevant lines from production.log
seem to be:
Started PUT "/<user>/lfstest.git/gitlab-lfs/objects/b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531/6194/authorize" for <IP address> at 2022-06-17 15:36:45 +0000
Processing by Repositories::LfsStorageController#upload_authorize as HTML
Parameters: {"repository_path"=>"<user>/lfstest.git", "oid"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "size"=>"6194"}
Completed 200 OK in 418ms (Views: 1.0ms | ActiveRecord: 5.0ms | Elasticsearch: 0.0ms | Allocations: 47529)
Started PUT "/<user>/lfstest.git/gitlab-lfs/objects/b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531/6194" for <IP address> at 2022-06-17 15:36:45 +0000
Processing by Repositories::LfsStorageController#upload_finalize as HTML
Parameters: {"file.gitlab-workhorse-upload"=>"<...>", "file.md5"=>"<...>", "file.name"=>"", "file.path"=>"", "file.remote_id"=>"1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0", "file.remote_url"=>"https://s3.<region>.amazonaws.com/<bucket_name>/gitlab-lfs-objects/tmp/uploads/1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0?X-Amz-Expires=15300&X-Amz-Date=20220617T153645Z&X-Amz-Security-Token=<...>&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<...>&X-Amz-SignedHeaders=host&X-Amz-Signature=<...>", "file.sha1"=>"<...>", "file.sha256"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "file.sha512"=>"<...>", "file.size"=>"6194", "file.upload_duration"=>"0.018153652", "file"=>#<UploadedFile:0x00007f9318c994d0 @size=6194, @upload_duration=0.018153652, @content_type="application/octet-stream", @original_filename="unnamed", @sha256="b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", @remote_id="1655480205-274883-0001-0492-e98dd52ebfcf5f71cc1fbb080c07fac0">, "repository_path"=>"<user>/lfstest.git", "oid"=>"b043b67e1953f40d75a16316eec4b5e9d5f2659c83f90ed7db72a6fbd2f03531", "size"=>"6194"}
Completed 403 Forbidden in 108ms (Views: 0.3ms | ActiveRecord: 6.3ms | Elasticsearch: 0.0ms | Allocations: 19319)
Similar lines are repeated several times as the git-lfs client retries.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 20.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.6 Sidekiq Version:6.4.0 Go Version: unknown GitLab information Version: 15.0.3-ee Revision: b7e551ef451 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.6 URL: http://gitlab.ourdomain.com HTTP Clone URL: http://gitlab.ourdomain.com/some-group/some-project.git SSH Clone URL: git@gitlab.ourdomain.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell