Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced
Summary
Reproduced on GitLab 15.11.0-ee
When cloning a git repository containing LFS objects from a secondary site, the git-lfs
client actually
downloads the LFS objects from the primary site. This happens despite the secondary site
reporting that all LFS objects are fully synced via the gitlab-rake geo:status
command.
Do note that unified URL is not enabled and each site has a separate URL. This is a standard Geo environment with 1 all-in-one node in the primary and secondary sites.
No feature flags have been set on the sites either.
Steps to reproduce
-
On the primary site, create a project containing some LFS objects
-
Check that the project is replicated to the secondary site, and that all LFS objects are synced with the
gitlab-rake geo:status
command -
Clone the project from the secondary site
GIT_TRACE=1 GIT_CURL_VERBOSE=1 git clone git@secondary.site:group/project.git
-
The output will show the initial request to the secondary site:
08:56:55.343509 trace git-lfs: HTTP: POST https://secondary.site/group/project.git/info/lfs/objects/batch
The next HTTP requests are then sent to the primary site to download the LFS objects:
08:56:56.729612 trace git-lfs: HTTP: GET https://primary.site/group/project.git/gitlab-lfs/objects/4ac46772c0465308e166ca215ceada891609d15f618996bdf67252d241fb85e6
Example Project
N/A
What is the current bug behavior?
When cloning a git repository containing LFS objects from a secondary site and the secondary site is up to date, Workhorse proxies the HTTP: POST https://secondary.site/group/project.git/info/lfs/objects/batch
request back to primary site. This results in the LFS objects being downloaded from the primary site.
What is the expected correct behavior?
When cloning a git repository containing LFS objects from a secondary site and the secondary site is up to date, Workhorse should not proxy the request and the LFS objects should be downloaded from the secondary site.
Relevant logs and/or screenshots
-
Logs from the
git clone
: geo-lfs-proxied-clone.txt -
If we look for the correlation ID shown in the
git clone
toPOST https://secondary.site/group/project.git/info/lfs/objects/batch
, we don't find anything on the secondary site. Instead, we find the logs on the primary site, which confirms that the request is proxied via Workhorse:# On primary site $ rg 01H014V2VEZNVR9X69EE30KECF gitlab-workhorse/current 9764:{"content_type":"application/vnd.git-lfs+json; charset=utf-8","correlation_id":"01H014V2VEZNVR9X69EE30KECF","duration_ms":201,"host":"primary.site","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"<sanitized>:0","remote_ip":"<sanitized>","route":"","status":200,"system":"http","time":"2023-05-09T20:56:56Z","ttfb_ms":201,"uri":"/group/project.git/info/lfs/objects/batch","user_agent":"git-lfs/3.3.0 (GitHub; darwin arm64; go 1.19.3)","written_bytes":6200} gitlab-rails/production_json.log 4720:{"method":"POST","path":"/group/project.git/info/lfs/objects/batch","format":"json","controller":"Repositories::LfsApiController","action":"batch","status":200,"time":"2023-05-09T20:56:56.629Z","params":[{"key":"operation","value":"download"},{"key":"objects","value":[{"oid":"4ac46772c0465308e166ca215ceada891609d15f618996bdf67252d241fb85e6","size":26692892},{"oid":"bfd79adc63a7348814b55a80ef6c16beee801e4b92bdfdbbb7c5fb9837629df6","size":8657380},{"oid":"c9b099d083664983b2de3509a2753502097fc7981a65f8a35680fa9be326f5f4","size":8263432},{"oid":"1d3ce056b37a24d2e58815801f035c972016779b566262323e385e0569488455","size":5507382},{"oid":"789454f530aa3369fa8f11d73450aaea7749b2f1618df6178f09eead6f2cde07","size":3524272},{"oid":"20ef06dd0b2a6bcbe515675d1a4bf9b6ef9a70bd2e57c00133b872c2ef8f21e4","size":2651013},{"oid":"55f2c03f6c6363c219dd18690a24cbea64623ef8ae4a74921d3f43c61dc1e9cf","size":362062},{"oid":"89c76fd41d7e42ce0a7c37a85b108c5a941bb3819bbcc70dddb62ef026f61300","size":362057},{"oid":"dc17e835a4015c31c6d9a2609194960178587afa904be9e0013ce003e1f3cee7","size":358667},{"oid":"616de117701793d7a109101dbd3bcb6e6abc10a6b03c9431817df399ee952650","size":356968}]},{"key":"transfers","value":["lfs-standalone-file","basic","ssh"]},{"key":"ref","value":{"name":"refs/heads/master"}},{"key":"hash_algo","value":"sha256"},{"key":"repository_path","value":"group/project.git"},{"key":"lfs_api","value":{"operation":"download","objects":[{"oid":"4ac46772c0465308e166ca215ceada891609d15f618996bdf67252d241fb85e6","size":26692892},{"oid":"bfd79adc63a7348814b55a80ef6c16beee801e4b92bdfdbbb7c5fb9837629df6","size":8657380},{"oid":"c9b099d083664983b2de3509a2753502097fc7981a65f8a35680fa9be326f5f4","size":8263432},{"oid":"1d3ce056b37a24d2e58815801f035c972016779b566262323e385e0569488455","size":5507382},{"oid":"789454f530aa3369fa8f11d73450aaea7749b2f1618df6178f09eead6f2cde07","size":3524272},{"oid":"20ef06dd0b2a6bcbe515675d1a4bf9b6ef9a70bd2e57c00133b872c2ef8f21e4","size":2651013},{"oid":"55f2c03f6c6363c219dd18690a24cbea64623ef8ae4a74921d3f43c61dc1e9cf","size":362062},{"oid":"89c76fd41d7e42ce0a7c37a85b108c5a941bb3819bbcc70dddb62ef026f61300","size":362057},{"oid":"dc17e835a4015c31c6d9a2609194960178587afa904be9e0013ce003e1f3cee7","size":358667},{"oid":"616de117701793d7a109101dbd3bcb6e6abc10a6b03c9431817df399ee952650","size":356968}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/master"},"hash_algo":"sha256"}}],"correlation_id":"01H014V2VEZNVR9X69EE30KECF","meta.caller_id":"Repositories::LfsApiController#batch","meta.remote_ip":"<sanitized>","meta.feature_category":"source_code_management","meta.user":"anton","meta.user_id":1,"meta.project":"group/project","meta.root_namespace":"group","meta.client_id":"user/1","remote_ip":"<sanitized>","user_id":1,"username":"anton","ua":"git-lfs/3.3.0 (GitHub; darwin arm64; go 1.19.3)","queue_duration_s":0.023383,"request_urgency":"medium","target_duration_s":0.5,"redis_calls":14,"redis_duration_s":0.008714,"redis_read_bytes":1020,"redis_write_bytes":1569,"redis_cache_calls":11,"redis_cache_duration_s":0.008024,"redis_cache_read_bytes":1018,"redis_cache_write_bytes":1393,"redis_rate_limiting_calls":3,"redis_rate_limiting_duration_s":0.00069,"redis_rate_limiting_read_bytes":2,"redis_rate_limiting_write_bytes":176,"db_count":12,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_main_count":12,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.022,"db_main_duration_s":0.022,"db_main_replica_duration_s":0.0,"cpu_s":0.095673,"mem_objects":45299,"mem_bytes":9269032,"mem_mallocs":34152,"mem_total_bytes":11080992,"pid":17598,"worker_id":"puma_0","rate_limiting_gates":[],"db_duration_s":0.08098,"view_duration_s":0.00049,"duration_s":0.15365}
-
Sync status on the secondary site:
# On secondary site $ sudo gitlab-rake geo:status Name: secondary URL: https://secondary.site ----------------------------------------------------- GitLab Version: 15.11.0-ee Geo Role: Secondary Health Status: Healthy Repositories: succeeded 7 / total 7 (100%) Verified Repositories: succeeded 7 / total 7 (100%) Design repositories: succeeded 0 / total 0 (0%) Lfs Objects: succeeded 10 / total 10 (100%) Merge Request Diffs: succeeded 0 / total 0 (0%) Package Files: succeeded 0 / total 0 (0%) Terraform State Versions: succeeded 0 / total 0 (0%) Snippet Repositories: succeeded 2 / total 2 (100%) Group Wiki Repositories: succeeded 0 / total 0 (0%) Pipeline Artifacts: succeeded 0 / total 0 (0%) Pages Deployments: succeeded 0 / total 0 (0%) Uploads: succeeded 2 / total 2 (100%) Job Artifacts: succeeded 0 / total 0 (0%) Ci Secure Files: succeeded 0 / total 0 (0%) Dependency Proxy Blobs: succeeded 0 / total 0 (0%) Dependency Proxy Manifests: succeeded 0 / total 0 (0%) Project Wiki Repositories: succeeded 7 / total 7 (100%) Repositories Checked: succeeded 6 / total 7 (85%) Lfs Objects Verified: succeeded 10 / total 10 (100%) Merge Request Diffs Verified: succeeded 0 / total 0 (0%) Package Files Verified: succeeded 0 / total 0 (0%) Terraform State Versions Verified: succeeded 0 / total 0 (0%) Snippet Repositories Verified: succeeded 2 / total 2 (100%) Pipeline Artifacts Verified: succeeded 0 / total 0 (0%) Pages Deployments Verified: succeeded 0 / total 0 (0%) Uploads Verified: succeeded 2 / total 2 (100%) Job Artifacts Verified: succeeded 0 / total 0 (0%) Ci Secure Files Verified: succeeded 0 / total 0 (0%) Dependency Proxy Blobs Verified: succeeded 0 / total 0 (0%) Dependency Proxy Manifests Verified: succeeded 0 / total 0 (0%) Project Wiki Repositories Verified: succeeded 7 / total 7 (100%) Sync Settings: Full Database replication lag: 0 seconds Last event ID seen from primary: 186 (about 17 hours ago) Last event ID processed by cursor: 186 (about 17 hours ago) Last status report was: 2 minutes ago
-
gitlab.rb
configuration (passwords have been removed):ubuntu@primary:~$ sudo grep -Ev "password|_key|token|secret|app_id|bind_dn|^$|^\s*#" /etc/gitlab/gitlab.rb external_url 'https://primary.site' roles ['geo_primary_role'] gitlab_rails['auto_migrate'] = true postgresql['listen_address'] = "0.0.0.0" postgresql['md5_auth_cidr_addresses'] = ["127.0.0.1/32", "x.x.x.x/32", "x.x.x.x/32"] gitlab_rails['geo_node_name'] = "primary" ubuntu@secondary:~$ sudo grep -Ev "password|_key|token|secret|app_id|bind_dn|^$|^\s*#" /etc/gitlab/gitlab.rb external_url 'https://secondary.site' roles ['geo_secondary_role'] postgresql['listen_address'] = "0.0.0.0" postgresql['md5_auth_cidr_addresses'] = ["127.0.0.1/32", "x.x.x.x/32"] gitlab_rails['geo_node_name'] = "secondary"
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)