Mutual-TLS (mTLS) support for fetching LFS objects
Problem to solve
Currently we don't seem to have a documented and/or supported way to allow GitLab to fetch LFS objects using Mutual-TLS (mTLS). This result of this is that the following example scenarios are not currently possible:
- Attempting to fetch LFS objects from an environment (such as another GitLab instance), where mTLS is enforced.
- Repository mirroring inclusive of LFS objects from an environment (such as another GitLab instance), where mTLS is enforced.
This was encountered by a customer that was trying to setup repository import + mirroring, where the source GitLab environment is currently using mTLS / 2-way SSL client authentication.
Upon trying to get this working, any repositories with LFS objects aren't able to get those LFS objects across into the source environment.
Upon further discussion, it was clarified that Sidekiq will be performing the LFS fetch, however it seems that there is no documented way for Sidekiq to serve a client TLS certificate (for the purpose of mTLS). In this case, the source GitLab environment will not allow the LFS object fetch attempts as these connections aren't passing mTLS.
There is currently another open issue that has relevance - Repository mirroring using client certificate authentication, where "full" support of repository mirroring using mTLS should include LFS objects to be considered a complete feature.
Example scenario
Overview
- A customer needs to import and mirror projects from one GitLab environment to another, for sake of example let's say -
GitLab Environment A
toGitLab Environment B
. - Both environments use S3 object storage for LFS objects with
proxy_download
enabled - so the objects are not directly fetched from S3, instead the connections go directly in/out of the GitLab environment nodes. - The environments need to use mutual TLS.
- There will be repositories involved that contain LFS objects.
The repository import process - Repository content (excluding LFS objects)
-
While not documented officially at this point in time, it seems to be possible to get
GitLab Environment B
to deal with mutual TLS and supply a certificate via this config within the Gitaly node(s)gitlab.rb
file - similar to this suggestion:gitaly['gitconfig'] = [ { 'section': 'http', 'key': 'sslVerify', 'value:': 'true' }, { 'section': 'http', 'key': 'sslCert', 'value:': '/var/opt/gitlab/example-client-cert.crt' }, { 'section': 'http', 'key': 'sslKey', 'value:': '/var/opt/gitlab/example-client-cert.key' }, ]
-
Gitaly uses that configuration to successfully connect to the destination server via mutual TLS and pull in the git repo content (minus LFS objects).
The repository import process - LFS objects
- Per what was discussed in Document how LFS objects are fetched by GitLab during an import or repository pull mirror operation, it seems Sidekiq is what tries to fetch the LFS objects - but it appears there is no documented way to configure Sidekiq to use mutual TLS / define a client cert for it to use. In which case the attempts to fetch LFS objects as part of an repository import or mirroring operation, cannot succeed.
Steps to reproduce
-
There will need to be two separate GitLab environments for testing this process. Two single node instances should suffice.
- A "source" environment. This will be the initial environment that stores the repositories containing LFS objects.
- A "destination" environment. This will be the environment that we intend to import the repositories into. The end goal will be that both of these environments mirror the repositories back and forth to eachother.
-
Create a project on the "source" GitLab environment where the repository contains LFS objects. Alternatively, just import this project - https://gitlab.com/jamesreed/lfs_test
-
Enable mTLS / 2-way SSL client authentication on what will be a "source" GitLab environment that originally stores our repositories that contain LFS objects.
-
On the "destination" GitLab environment, setup Gitaly so that it can serve the client TLS certificate. Within the
gitlab.rb
- this can be done with something like the example in this suggestion:gitaly['gitconfig'] = [ { 'section': 'http', 'key': 'sslVerify', 'value:': 'true' }, { 'section': 'http', 'key': 'sslCert', 'value:': '/var/opt/gitlab/example-client-cert.crt' }, { 'section': 'http', 'key': 'sslKey', 'value:': '/var/opt/gitlab/example-client-cert.key' }, ]
-
Attempt to import the project from the "source" environment to the "destination" environment. In the UI it should look as though the import has worked, however if we click on any of the LFS objects within the repository in the GitLab UI and try to download them, a
404
will be generated. -
Running the following script against the project on the "source" environment for the imported project will reveal that no LFS objects exist:
Project.find(<project-id>).lfs_objects.each do |lo| puts "#{lo.id},#{lo.file.file.exists?},#{lo.file.file.path}" end
Current behavior
LFS objects are not fetched during repository import and/or mirroring operations, where the instance the objects are pulled from is configured to use mTLS.
Expected behavior
Allow for Sidekiq to provide a client certificate for mTLS connections, so that LFS objects are supported during the import and/or mirroring process.
Versions
This was tested on:
- Source environment:
16.2.3-ee
- Destination environment:
16.2.3-ee
Relevant logs
We aren't seeing particularly verbose logging from GitLab itself for when the LFS object fetch attempts fail in this scenario.
-
What we do see on the destination environment side is error output from Sidekiq like the below example from
var/log/gitlab/gitlab-rails/update_mirror_service_json.log
:{"severity":"ERROR","time":"2023-08-25T23:44:07.598Z","correlation_id":"26bd7fd8e5f53de8249c04c6c7930100","user_id":1,"project_id":34,"import_url":"https://*****:*****@gitlab-latest-v16.gitlab.example.com/root/bigfile-test.git","error_message":"The LFS objects download list couldn't be imported. Error: Bad Request"}
- This seems to be generated when Sidekiq is trying to get the list of LFS objects to mirror - https://gitlab.com/gitlab-org/gitlab/-/blob/v15.8.4-ee/app/services/projects/lfs_pointers/lfs_object_download_list_service.rb?ref_type=tags#L26
- If this understanding is correct, then the failures are happening when grabbing the list of LFS objects to fetch before the actual "LFS object fetch" even has a chance to occur, so attempting to work around this with something like configuring the environments to use non-proxied object storage for LFS objects (for example, so that LFS objects fetches end up using direct AWS S3 URLs, where mTLS would not necessarily apply), may not be possible.
- This seems to be generated when Sidekiq is trying to get the list of LFS objects to mirror - https://gitlab.com/gitlab-org/gitlab/-/blob/v15.8.4-ee/app/services/projects/lfs_pointers/lfs_object_download_list_service.rb?ref_type=tags#L26
-
Some of the error logging surrounding this process may be improvable once the following issue is addressed: Better error messages for Pull Mirroring errors