Support for mirroring via SSH
Summary
When deployed via the helm chart, repository mirroring via SSH fails because the ssh
binary is not present in the Sidekiq container.
The ssh-keyscan
binary is also not present which, I think, will break host key detection.
Steps to reproduce
- Deploy GitLab and create a project
- Configure pull mirroring of a repository with a ssh:// url
Configuration used
See configuration snippet in #1022 (closed)
Current behavior
Mirroring fails with the logs attached to the bottom of this issue. To check this I kubectl exec
-ed into the sidekiq container and confirmed that ssh
is not on the PATH
.
Expected behavior
Repositories may be mirrored over SSH.
Versions
GitLab: 11.7.5-ee Chart: 1.5.3
Versions of other components available on request.
Relevant logs
Result of running kubectl exec
on sidekiq container demonstrating lack of ssh binary on PATH
.
$ kubectl -n REDACTED exec -it REDACTED-sidekiq-all-in-1-REDACTED -- /bin/sh -c 'id'
uid=1000(git) gid=1000(git) groups=1000(git)
$ kubectl -n REDACTED exec -it REDACTED-sidekiq-all-in-1-REDACTED -- /bin/sh -c 'ssh'
/bin/sh: 1: ssh: not found
command terminated with exit code 127
Sidekiq log
I Mirror update for uis/devops/infra/ansible/infra started. Waiting duration: 0
I Mirror update for uis/devops/infra/ansible/infra failed with the following message: 2:Fetching remote upstream failed: ssh '-oIdentityFile="/tmp/gitlab-shell-key-file20190220-24-18l8vvq"' '-oIdentitiesOnly="yes"' '-oStrictHostKeyChecking="yes"' '-oUserKnownHostsFile="/tmp/gitlab-shell-known-hosts20190220-24-1y3qt0n"': 1: ssh '-oIdentityFile="/tmp/gitlab-shell-key-file20190220-24-18l8vvq"' '-oIdentitiesOnly="yes"' '-oStrictHostKeyChecking="yes"' '-oUserKnownHostsFile="/tmp/gitlab-shell-known-hosts20190220-24-1y3qt0n"': ssh: not found
I fatal: Could not read from remote repository.
I
I Please make sure you have the correct access rights
I and the repository exists.
I
I 2019-02-20T15:25:17.018Z 9 TID-gsue0cmmt Geo::SidekiqCronConfigWorker JID-15bdefd8793af25ec44957e2 INFO: start
I 2019-02-20T15:25:17.063Z 9 TID-gsuf7clf9 RepositoryUpdateMirrorWorker JID-6387109e9d26fca6f0b463f5 INFO: fail: 0.209 sec
I 2019-02-20T15:25:17.073Z 9 TID-gsuf7clf9 WARN: {"context":"Job raised exception","job":{"class":"RepositoryUpdateMirrorWorker","args":[140],"retry":false,"queue":"repository_update_mirror","status_expiration":54000,"jid":"6387109e9d26fca6f0b463f5","created_at":1550676316.852233,"correlation_id":"5yibRAcVdVa","enqueued_at":1550676316.8526285},"jobstr":"{\"class\":\"RepositoryUpdateMirrorWorker\",\"args\":[140],\"retry\":false,\"queue\":\"repository_update_mirror\",\"status_expiration\":54000,\"jid\":\"6387109e9d26fca6f0b463f5\",\"created_at\":1550676316.852233,\"correlation_id\":\"5yibRAcVdVa\",\"enqueued_at\":1550676316.8526285}"}
I 2019-02-20T15:25:17.100Z 9 TID-gsuf7clf9 WARN: RepositoryUpdateMirrorWorker::UpdateError: 2:Fetching remote upstream failed: ssh '-oIdentityFile="/tmp/gitlab-shell-key-file20190220-24-18l8vvq"' '-oIdentitiesOnly="yes"' '-oStrictHostKeyChecking="yes"' '-oUserKnownHostsFile="/tmp/gitlab-shell-known-hosts20190220-24-1y3qt0n"': 1: ssh '-oIdentityFile="/tmp/gitlab-shell-key-file20190220-24-18l8vvq"' '-oIdentitiesOnly="yes"' '-oStrictHostKeyChecking="yes"' '-oUserKnownHostsFile="/tmp/gitlab-shell-known-hosts20190220-24-1y3qt0n"': ssh: not found
I fatal: Could not read from remote repository.
I
I Please make sure you have the correct access rights
I and the repository exists.
I
I 2019-02-20T15:25:17.100Z 9 TID-gsuf7clf9 WARN: /srv/gitlab/ee/app/workers/repository_update_mirror_worker.rb:26:in `perform'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:185:in `execute_job'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:167:in `block (2 levels) in process'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
I /srv/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:15:in `block in call'
I /srv/gitlab/lib/gitlab/metrics/transaction.rb:55:in `run'
I /srv/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:15:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call'
I /srv/gitlab/lib/gitlab/correlation_id.rb:15:in `use_id'
I /srv/gitlab/lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/lib/gitlab/sidekiq_middleware/shutdown.rb:54:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sentry-raven-2.7.4/lib/raven/integrations/sidekiq.rb:9:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:166:in `block in process'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:108:in `local'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:42:in `block in call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/execution_wrapper.rb:85:in `wrap'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/reloader.rb:68:in `block in wrap'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/execution_wrapper.rb:85:in `wrap'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/reloader.rb:67:in `wrap'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:41:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:243:in `stats'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_logger.rb:8:in `call'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:73:in `global'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:125:in `block in dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/logging.rb:48:in `with_context'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:124:in `dispatch'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:165:in `process'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:83:in `process_one'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:71:in `run'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/util.rb:16:in `watchdog'
I /srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/util.rb:25:in `block in safe_thread'
Edited by Joshua Lambert