Geo + CNH: GitLab Shell secret copy requires deployment restart to take effect
During a Geo Secondary deployment, https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/ansible/roles/gitlab_geo/tasks/copy-secrets.yml is run to make sure the application secrets match across Geo Nodes (Primary and Secondary).
One of these secrets is gitlab-gitlab-shell-host-keys
, which defines the host key for the Geo Node when the instance is accessed via SSH.
But because gitlab-shell
loads its configuration on Pod
initialisation, the copied secret isn't assumed until after the deployment
of gitlab-shell
is restarted.
In this example, I'm calling SSH on a Primary and Secondary after running the Geo configuration:
❯ ssh git@itestshgitlabsshd.neigh-bor.link
The authenticity of host 'itestshgitlabsshd.neigh-bor.link (54.217.95.94)' can't be established.
ED25519 key fingerprint is SHA256:nh7nl4LqeKJbby+rooWBI7kDze+TnTQOORMxu62qfJw. This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'itestshgitlabsshd.neigh-bor.link' (ED25519) to the list of known hosts.
git@itestshgitlabsshd.neigh-bor.link: Permission denied (publickey,keyboard-interactive).
❯ ssh git@eu-west-2.itestshgitlabsshd.neigh-bor.link
The authenticity of host 'eu-west-2.itestshgitlabsshd.neigh-bor.link (18.175.13.76)' can't be established.
ED25519 key fingerprint is SHA256:Sl7X+fY15rPF+Hui1fBajY+bQ6rSuVV2QwTbzH7lbFQ. This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'eu-west-2.itestshgitlabsshd.neigh-bor.link' (ED25519) to the list of known hosts.
git@eu-west-2.itestshgitlabsshd.neigh-bor.link: Permission denied (publickey,keyboard-interactive).
Notice that the fingerprints for both hosts are different when they should have been the same because of the secret copy. I verified that the secret does get copied, but gitlab-shell
never restarted so it's still operating with the "old" (auto-generated) key.
Once I restart the eu-west-2
Deployment
the fingerprints match:
❯ ssh git@eu-west-2.itestshgitlabsshd.neigh-bor.link
The authenticity of host 'eu-west-2.itestshgitlabsshd.neigh-bor.link (18.132.122.239)' can't be established.
ED25519 key fingerprint is SHA256:nh7nl4LqeKJbby+rooWBI7kDze+TnTQOORMxu62qfJw.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:62: itestshgitlabsshd.neigh-bor.link
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'eu-west-2.itestshgitlabsshd.neigh-bor.link' (ED25519) to the list of known hosts.
git@eu-west-2.itestshgitlabsshd.neigh-bor.link: Permission denied (publickey,keyboard-interactive).
I think GET should reload the gitlab-shell
deployment after copying the host secret, but I am open to discussing different options.