Git clone on secondary Geo node fetches LFS files from Primary
Summary
When cloning a project from a secondary node that contains LFS files, the LFS files will be redirected to the primary and downloaded from there. This is likely as regression as this was already described in #7419 (closed) and fixed with !7209 (merged)
I did not encounter this behaviour on GitLab version 12.2.1
.
Steps to reproduce
Set .gitconfig
:
[filter "lfs"]
required = true
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
process = git-lfs filter-process
- Create a Geo primary and linked secondary with version
12.5.0
- Create a new project on the Geo primary e.g.
lfs-test
- Enable LFS and add an LFS file to
lfs-test
- Cloning the project on the secondary
When cloning this happens:
16:59:14.614229 trace git-lfs: api: batch 1 files
16:59:14.661139 trace git-lfs: HTTP: POST http://fzgeo-demo-primary-geo.gogitlab.ml/lfs-fork-test/lfs-wallpaper-test.git/info/lfs/objects/batch
> POST /lfs-fork-test/lfs-wallpaper-test.git/info/lfs/objects/batch HTTP/1.1
> Host: fzgeo-demo-primary-geo.gogitlab.ml
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 161
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
{"operation":"download","objects":[{"oid":"1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","size":3315954}],"ref":{"name":"refs/heads/master"}}16:59:15.051909 trace git-lfs: HTTP: 200
Example Project
I can provide a Geo instance that exhibits the behaviour.
What is the current bug behavior?
LFS read-only operations e.g. git clone
are not run against the secondary but are proxied to the primary
What is the expected correct behavior?
Read-only operations are run against the secondary.
Relevant logs and/or screenshots
Output of GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone
> env GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone git@fzgeo-demo-secondary-geo.gogitlab.ml:lfs-fork-test/lfs-wallpaper-test.git lfs-wallpaper-test-secondary Tue 26 Nov 16:59:00 2019
16:59:07.487299 git.c:439 trace: built-in: git clone git@fzgeo-demo-secondary-geo.gogitlab.ml:lfs-fork-test/lfs-wallpaper-test.git lfs-wallpaper-test-secondary
Cloning into 'lfs-wallpaper-test-secondary'...
16:59:07.495041 run-command.c:663 trace: run_command: unset GIT_DIR; ssh git@fzgeo-demo-secondary-geo.gogitlab.ml 'git-upload-pack '\''lfs-fork-test/lfs-wallpaper-test.git'\'''
Warning: the ECDSA host key for 'fzgeo-demo-secondary-geo.gogitlab.ml' differs from the key for the IP address '35.235.45.93'
Offending key for IP in /Users/fabian/.ssh/known_hosts:24
Matching host key in /Users/fabian/.ssh/known_hosts:37
Are you sure you want to continue connecting (yes/no)? yes
16:59:10.726299 run-command.c:663 trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 26157 on FZGL.fritz.box' --check-self-contained-and-connected
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (10/10), done.
16:59:10.737346 git.c:439 trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 26157 on FZGL.fritz.box' --check-self-contained-and-connected
remote: Total 13 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (13/13), 3.15 MiB | 7.29 MiB/s, done.
Resolving deltas: 100% (2/2), done.
16:59:11.361706 run-command.c:663 trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
16:59:11.367459 git.c:439 trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
16:59:11.373643 run-command.c:663 trace: run_command: 'git-lfs filter-process'
16:59:11.384689 trace git-lfs: exec: git 'version'
16:59:11.395221 trace git-lfs: exec: git 'config' '-l'
16:59:11.400320 trace git-lfs: Install hook: pre-push, force=false, path=/Users/fabian/Projects/lfs-wallpaper-test-secondary/.git/hooks/pre-push
16:59:11.400716 trace git-lfs: Install hook: post-checkout, force=false, path=/Users/fabian/Projects/lfs-wallpaper-test-secondary/.git/hooks/post-checkout
16:59:11.401041 trace git-lfs: Install hook: post-commit, force=false, path=/Users/fabian/Projects/lfs-wallpaper-test-secondary/.git/hooks/post-commit
16:59:11.401249 trace git-lfs: Install hook: post-merge, force=false, path=/Users/fabian/Projects/lfs-wallpaper-test-secondary/.git/hooks/post-merge
16:59:11.401389 trace git-lfs: Initialize filter-process
16:59:11.401564 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:59:11.406981 trace git-lfs: tq: running as batched queue, batch size of 100
16:59:11.407332 trace git-lfs: filepathfilter: accepting "claudio-schwarz-purzlbaum-Grg-t0TtJKg-unsplash.jpg"
16:59:11.407497 trace git-lfs: tq: sending batch of size 1
16:59:11.407635 trace git-lfs: run_command: ssh -- git@fzgeo-demo-secondary-geo.gogitlab.ml git-lfs-authenticate lfs-fork-test/lfs-wallpaper-test.git download
Warning: the ECDSA host key for 'fzgeo-demo-secondary-geo.gogitlab.ml' differs from the key for the IP address '35.235.45.93'
Offending key for IP in /Users/fabian/.ssh/known_hosts:24
Matching host key in /Users/fabian/.ssh/known_hosts:37
Are you sure you want to continue connecting (yes/no)? yes
16:59:14.614229 trace git-lfs: api: batch 1 files
16:59:14.661139 trace git-lfs: HTTP: POST http://fzgeo-demo-primary-geo.gogitlab.ml/lfs-fork-test/lfs-wallpaper-test.git/info/lfs/objects/batch
> POST /lfs-fork-test/lfs-wallpaper-test.git/info/lfs/objects/batch HTTP/1.1
> Host: fzgeo-demo-primary-geo.gogitlab.ml
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 161
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
{"operation":"download","objects":[{"oid":"1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","size":3315954}],"ref":{"name":"refs/heads/master"}}16:59:15.051909 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 672
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Tue, 26 Nov 2019 15:59:14 GMT
< Etag: W/"dfab07ae2e997ac57a87b4ce99819f75"
< Page-Title: GitLab
< Referrer-Policy: strict-origin-when-cross-origin
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx
< Set-Cookie: experimentation_subject_id=IjIxMGMxYWQ1LTgxMWEtNGNkYy1iNThlLTRiODUyNTFiZGUxMyI%3D--a81c8769c988e25cb2e5cd37921c9bdb64b79dda; domain=.gogitlab.ml; path=/; expires=Sat, 26 Nov 2039 15:59:14 -0000; HttpOnly
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: SCMjEmSnbq5
< X-Runtime: 0.066629
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
16:59:15.052172 trace git-lfs: HTTP: {"objects":[{"oid":"1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","size":3315954,"actions":{"download":{"href":"http://fzgeo-demo-primary-geo.gogitlab.ml/lfs-fork-test/lfs-wallpaper-test.git/gitlab-lfs/objects/1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","header":{"Authorization":"Basic ZmFiaWFuOmV5SmhiR2NpT2lKSVV6STFOaUlzSW5SNWNDSTZJa3BYVkNKOS5leUprWVhSaElqcDdJbUZqZEc5eUlqb2labUZpYVdGdUluMHNJbXAwYVNJNklqZzJOVGxtT1RobUxUbGhNMlV0TkRoa01DMWlNelZsTFRVNE56VmhZell4WXpaaFp
{"objects":[{"oid":"1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","size":3315954,"actions":{"download":{"href":"http://fzgeo-demo-primary-geo.gogitlab.ml/lfs-fork-test/lfs-wallpaper-test.git/gitlab-lfs/objects/1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54","header":{"Authorization":"Basic ZmFiaWFuOmV5SmhiR2NpT2lKSVV6STFOaUlzSW5SNWNDSTZJa3BYVkNKOS5leUprWVhSaElqcDdJbUZqZEc5eUlqb2labUZpYVdGdUluMHNJbXAwYVNJNklqZzJOVGxtT1RobUxUbGhNMlV0TkRoa01DMWlNelZsTFRVNE56VmhZell4WXpaaFp16:59:15.052452 trace git-lfs: HTTP: DSXNJbWxoZENJNk1UVTNORGM0TXprMU5Dd2libUptSWpveE5UYzBOemd6T1RRNUxDSmxlSEFpT2pFMU56UTNPRFUzTlRSOS45RkhublIzVjhYUXptRmp0ZEVXMUtsQjlselgtMkZpSWd4ZTlSU2FycGNR"}}}}]}
DSXNJbWxoZENJNk1UVTNORGM0TXprMU5Dd2libUptSWpveE5UYzBOemd6T1RRNUxDSmxlSEFpT2pFMU56UTNPRFUzTlRSOS45RkhublIzVjhYUXptRmp0ZEVXMUtsQjlselgtMkZpSWd4ZTlSU2FycGNR"}}}}]}16:59:15.052616 trace git-lfs: tq: starting transfer adapter "basic"
16:59:15.052641 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
16:59:15.052678 trace git-lfs: xfer: adapter "basic" started
16:59:15.052684 trace git-lfs: xfer: adapter "basic" worker 1 starting
16:59:15.052730 trace git-lfs: xfer: adapter "basic" worker 5 starting
16:59:15.052725 trace git-lfs: xfer: adapter "basic" worker 4 starting
16:59:15.052768 trace git-lfs: xfer: adapter "basic" worker 2 starting
16:59:15.052765 trace git-lfs: xfer: adapter "basic" worker 0 starting
16:59:15.052785 trace git-lfs: xfer: adapter "basic" worker 3 starting
16:59:15.052788 trace git-lfs: xfer: adapter "basic" worker 6 starting
16:59:15.052699 trace git-lfs: xfer: adapter "basic" worker 7 starting
16:59:15.052731 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
16:59:15.052916 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
16:59:15.052960 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
16:59:15.053235 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
16:59:15.053103 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
16:59:15.052890 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
16:59:15.053353 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
16:59:15.052989 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54"
16:59:15.054613 trace git-lfs: HTTP: GET http://fzgeo-demo-primary-geo.gogitlab.ml/lfs-fork-test/lfs-wallpaper-test.git/gitlab-lfs/objects/1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54
> GET /lfs-fork-test/lfs-wallpaper-test.git/gitlab-lfs/objects/1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54 HTTP/1.1
> Host: fzgeo-demo-primary-geo.gogitlab.ml
> Authorization: Basic * * * * *
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
16:59:15.152163 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 3315954
< Accept-Ranges: bytes
< Cache-Control: private
< Connection: keep-alive
< Content-Disposition: attachment; filename="da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54"
< Content-Transfer-Encoding: binary
< Content-Type: application/octet-stream
< Date: Tue, 26 Nov 2019 15:59:15 GMT
< Last-Modified: Tue, 26 Nov 2019 15:58:46 GMT
< Referrer-Policy: strict-origin-when-cross-origin
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx
< Set-Cookie: experimentation_subject_id=IjdjZDJmNzI0LTJlZDgtNGZkNy05OTA3LTI0MTZiYzJkMWNjMCI%3D--0e05b07de12367853542840dfb93452f578a8bd4; domain=.gogitlab.ml; path=/; expires=Sat, 26 Nov 2039 15:59:15 -0000; HttpOnly
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: IlV8STmbRJ3
< X-Runtime: 0.047804
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
16:59:15.152767 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
16:59:15.152742 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
16:59:15.152837 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
16:59:15.152804 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
16:59:15.152811 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
16:59:15.152818 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
16:59:15.152794 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
16:59:18.406075 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "1471da982eb29a17ed224fe677d25f4b6d8f3e1e61b0170b32f9cd3e20ab2f54"
16:59:18.406105 trace git-lfs: xfer: adapter "basic" End()
16:59:18.406147 trace git-lfs: xfer: adapter "basic" worker 1 stopping
16:59:18.406152 trace git-lfs: xfer: adapter "basic" worker 6 stopping
16:59:18.406161 trace git-lfs: xfer: adapter "basic" worker 4 stopping
16:59:18.406168 trace git-lfs: xfer: adapter "basic" worker 5 stopping
16:59:18.406168 trace git-lfs: xfer: adapter "basic" worker 2 stopping
16:59:18.406170 trace git-lfs: xfer: adapter "basic" worker 0 stopping
16:59:18.406179 trace git-lfs: xfer: adapter "basic" worker 7 stopping
16:59:18.406177 trace git-lfs: xfer: adapter "basic" worker 3 stopping
16:59:18.406207 trace git-lfs: xfer: adapter "basic" stopped
16:59:18.407143 trace git-lfs: filepathfilter: accepting "claudio-schwarz-purzlbaum-Grg-t0TtJKg-unsplash.jpg"
16:59:18.417367 run-command.c:663 trace: run_command: /Users/fabian/Projects/lfs-wallpaper-test-secondary/.git/hooks/post-checkout 0000000000000000000000000000000000000000 63439daea512676422f60d6287f166ba12427222 1
16:59:18.582543 git.c:702 trace: exec: git-lfs post-checkout 0000000000000000000000000000000000000000 63439daea512676422f60d6287f166ba12427222 1
16:59:18.582994 run-command.c:663 trace: run_command: git-lfs post-checkout 0000000000000000000000000000000000000000 63439daea512676422f60d6287f166ba12427222 1
16:59:18.592525 trace git-lfs: exec: git 'version'
16:59:18.610189 trace git-lfs: exec: git 'config' '-l'
16:59:18.615048 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:59:18.620103 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
16:59:18.620126 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
16:59:18.620178 trace git-lfs: filepathfilter: accepting "lfs-wallpaper-test-secondary"
16:59:18.620318 trace git-lfs: filepathfilter: accepting "claudio-schwarz-purzlbaum-Grg-t0TtJKg-unsplash.jpg"
16:59:18.620334 trace git-lfs: filepathfilter: accepting "README.md"
16:59:18.620343 trace git-lfs: filepathfilter: accepting ".gitattributes"
16:59:18.620372 trace git-lfs: filepathfilter: rejecting ".git" via ".git"
16:59:18.621683 run-command.c:46 trace: run_command: running exit handler for pid 26161
16:59:18.621858 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
16:59:18.621876 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
16:59:18.621925 trace git-lfs: filepathfilter: accepting "tmp"
Results of GitLab environment info
Primary GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.5.0-ee Revision: 4878f9ac894 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: http://fzgeo-demo-primary-geo.gogitlab.ml HTTP Clone URL: http://fzgeo-demo-primary-geo.gogitlab.ml/some-group/some-project.git SSH Clone URL: git@fzgeo-demo-primary-geo.gogitlab.ml:some-group/some-project.git Elasticsearch: no Geo: yes Geo node: Primary Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 10.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Secondary GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.5.0-ee Revision: 4878f9ac894 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: http://fzgeo-demo-secondary-geo.gogitlab.ml HTTP Clone URL: http://fzgeo-demo-secondary-geo.gogitlab.ml/some-group/some-project.git SSH Clone URL: git@fzgeo-demo-secondary-geo.gogitlab.ml:some-group/some-project.git Elasticsearch: no Geo: yes Geo node: Secondary Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 10.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes
Secondary GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes
Possible fixes
!7209 (merged) may have some clues