Dependency proxy not fetching from object storage
Summary
The dependency proxy feature does not read from object storage when configured to do so.
If you configure object storage for the dependency proxy, the blobs in the database always have file_store
as null. This results in them being read from the fileystem, rather than from object storage.
Further, I've noticed that if direct_upload
is false, the objects are also not uploaded to object storage. With it set to true, they are uploaded, but not used during download.
Steps to reproduce
- Configure dependency proxy with object storage enabled
- Leave direct_upload set to true, to repro the fetch error
- Be sure to add EE license with proxy support to your instance
- Create a group to use with your proxy
- Pull an image you don't have locally from the proxy
- Check that new content was added to your bucket
- Open the rails console and list your DependencyProxy::Blob
- Note that file_store: is nil
Example failure on next image download, caught from the helm charts (which don't have shared storage other than object storage):
Completed 500 Internal Server Error in 309ms (ActiveRecord: 4.6ms | Elasticsearch: 0.0ms | Allocations: 7950)
gitlab-webservice-6848684fdf-7wbq5 webservice
gitlab-webservice-6848684fdf-7wbq5 webservice ActionController::MissingFile (Cannot read file /srv/gitlab/shared/dependency_proxy/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/dependency_proxy/2/files/1/323d0d660b6a7da8df08a01dbc7250f38cfa2161db00c7c27c0b20be07a8236a.gz):
gitlab-webservice-6848684fdf-7wbq5 webservice
gitlab-webservice-6848684fdf-7wbq5 webservice app/controllers/concerns/send_file_upload.rb:22:in `send_upload'
gitlab-webservice-6848684fdf-7wbq5 webservice ee/app/controllers/groups/dependency_proxy_for_containers_controller.rb:27:in `blob'
Example Project
Dependency Proxy MR being added to the helm charts encounters this error: gitlab-org/charts/gitlab!1464 (merged)
What is the current bug behavior?
Dependency proxy always records local file store to the database.
What is the expected correct behavior?
Dependency proxy should use the remote filestore when configured to do so.
Relevant logs and/or screenshots
My dependency proxy config:
dependency_proxy:
enabled: true
object_store:
enabled: true
remote_directory: <redacted>
direct_upload: true
background_upload: false
proxy_download: true
connection:
provider: AWS
region: us-east-1
aws_access_key_id: "<redacted>"
aws_secret_access_key: "<redacted>"
host: <redacted>
endpoint: <redacted>
path_style: true
Possible fixes
Likely, similar to the terraform state uploaders, we need to add a default store check, that uses the remote store when object storage is enabled. And also we need to have the file_store updated in the database after the store! method is fired. As seen in the terraform state model. These two things are missing from the dependency_proxy blob and uploader.