Mirroring via SSH Always Fails
When setting up a remote mirror to pull via SSH, the mirror will always fail.
The KnownHosts file contains entries tagged by the domain name and not the ip address.
ssh
by default configures CheckHostIP to yes
. The CheckHostIP option does not honor StrictHostKeyChecking by design to prevent the kind of attacks described above.
CheckHostIP
If set to yes (the default), ssh(1) will additionally check the host IP
address in the known_hosts file. This allows it to detect if a host key
changed due to DNS spoofing and will add addresses of destination hosts to
~/.ssh/known_hosts in the process, regardless of the setting of
StrictHostKeyChecking. If the option is set to no, the check will not be
executed.
Thus, when the attempt is made to connect using git
via ssh
the service will try to write to KnownHosts and that will fail. This failure causes the whole git
command to fail.
time="2019-03-13T21:13:30Z" level=error msg="finished unary call with code Unknown" correlation_id=iqzpYRQyRda error="rpc error: code = Unknown desc = Fetching remote upstream failed: Failed to add the ECDSA host key for IP address '35.231.145.151' to the list of known hosts (/tmp/gitlab-shell-known-hosts20190313-27-13werng).\r\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n" grpc.code=Unknown grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq grpc.method=FetchRemote grpc.request.fullMethod=/gitaly.RepositoryService/FetchRemote grpc.request.glProjectPath=root/awesomessh grpc.request.glRepository=project-1 grpc.request.repoPath=root/awesomessh.git grpc.request.repoStorage=default grpc.request.topLevelGroup=root grpc.service=gitaly.RepositoryService grpc.start_time="2019-03-13T21:13:29Z" grpc.time_ms=513.846 peer.address="10.20.0.24:53210" span.kind=server system=grpc
Originally Researched in: https://gitlab.com/charts/gitlab/issues/1167