Geo: Pull via SSH fails for non-replicated repo with multiple branches
Summary
This bug causes pull request that are not replicates/updated on the secondary node to fail if the repository has multiple branches
This only affects SSH pull requests and not HTTPS requests
This only affects Repositories with multiple branches and not Repositories with a single branch. If using --single-branch
flag the pull succeeds.
Error message returned is fatal: bad object 66504ff45e149df53714f9da4c1752cf8d1d4880
Steps to reproduce
The easiest way to replicate this consistently is to setup a secondary node using Selective Synchronisation
This has been replicated on a GEO setup with 1 Primary and 1 Secondary nodes with version 15.8.1.
- Setup a GEO environment with a single Primary.
- Setup Secondary with only selective groups for replication.
- Create a project with multiple branches in a group that is not part of the replicated groups.
- Pull this project via SSH.
What is the current bug behavior?
When pulling a non replicated repo or a repo that is not yet synchronised via a secondary GEO node the below error is received.
$ git clone git@geo-secondary.example.com:unsyncronised_group/multi_branch_project.git
Cloning into 'multi_branch_project'...
remote:
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
remote: git@geo-primary.example.com:unsyncronised_group/multi_branch_project.git
remote:
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
fatal: bad object 66504ff45e149df53714f9da4c1752cf8d1d4880
fatal: remote did not send all necessary objects
We can also verify that this object does exist on the primary node.
What is the expected correct behavior?
clone to succeed as when cloning directly from the primary:
$ git clone git@geo-primary.example.com:unsyncronised_group/multi_branch_project.git
Cloning into 'multi_branch_project'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), done.
Possible fixes
Possible workarounds depending on customer needs:
- If you don't have a need to use SSH you can instead use HTTPS
- If you do need to use SSH but only require the master/first branch you can use the flag
--single-branch
Related issues
The problem reported here is a result of translating the git over ssh protocol the the git over https protocol when proxying from the secondary site to the primary site. The work outlined the epic to remove the translation step is intended to provide a solution.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Primary GEO Node
$ sudo gitlab-rake gitlab:env:info System information System: Ubuntu 20.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.7p221 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.8 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.8.1-ee Revision: c49deff6e37 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://geo-primary.example.com HTTP Clone URL: https://geo-primary.example.com/unsyncronised_group/multi_branch_project.git SSH Clone URL: git@geo-primary.example.com:unsyncronised_group/multi_branch_project.git Elasticsearch: no Geo: yes Geo node: Primary Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.15.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Secondary GEO node
$ sudo gitlab-rake gitlab:env:info System information System: Ubuntu 20.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.7p221 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.8 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.8.1-ee Revision: c49deff6e37 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://geo-secondary.example.com HTTP Clone URL: https://geo-secondary.example.com/unsyncronised_group/multi_branch_project.git SSH Clone URL: git@geo-secondary.example.com:unsyncronised_group/multi_branch_project.git Elasticsearch: no Geo: yes Geo node: Secondary Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.15.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Primary GEO Node
$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.15.0 ? ... OK (14.15.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 4/2 ... yes 10/3 ... yes 9/4 ... yes 10/5 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.7) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo is available ... GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a primary node named "primary_node" HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... yes GitLab configured to disable writing to authorized_keys file ... no Try fixing it: You need to disable
Write to authorized_keys file
in GitLab's Admin panel For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yesChecking Geo ... Finished
Checking GitLab subtasks ... Finished
Secondary GEO Node
$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.15.0 ? ... OK (14.15.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 4/2 ... yes 10/3 ... yes 9/4 ... yes 10/5 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.7) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo secondary database is correctly configured ... yes Database replication enabled? ... yes Database replication working? ... yes GitLab Geo HTTP(S) connectivity ...
- Can connect to the primary node ... yes
GitLab Geo is available ...
GitLab Geo is enabled ... yes
This machine's Geo node name matches a database record ... yes, found a secondary node named "secondary_node"
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... no
Try fixing it:
You need to disable
Write to authorized_keys file
in GitLab's Admin panel For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking Geo ... Finished
Checking GitLab subtasks ... Finished
</pre>
</details>