SSH executor does not work as expected ...

Hello, I can't get SSH executor to work on my simple setup as explained below. Since yesterday, I have gone through all sorts of documentation, but could not find any information on why is this failing? Your help will be appreciated. Thanks.

I have the following setup:

[My repo on gitlab.com] --> Manual Runner --> gitlab-runner on my home server --> (:5422)[builder@192.168.0.54]

Other details:

My repo: https://gitlab.com/kamranazeem/development-on-shared-mac
Gitlab-runner runs as a docker container on Fedora Linux at my home - named 'pserver5'.
192.168.0.54 is a MAC computer, with a user builder, allowing key based access only over SSH - with /bin/bash set as default SHELL.

My gitlab-runner config file:

[root@pserver5 gitlab-runner]# cat /home/containers-data/gitlab-runner/config/config.toml

[[runners]]
  name = "macOS-ssh-runner"
  url = "https://gitlab.com/"
  token = "hQwrVU-XXXXXXXXXXXX"
  executor = "ssh"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.ssh]
    user = "builder"
    host = "192.168.0.54"
    port = "22"
    identity_file = "/home/containers-secrets/gitlab-runner/id_ed25519_builder"

The file exists and the permissions are as follows:

[root@pserver5 gitlab-runner]# pwd
/home/containers-secrets/gitlab-runner

[root@pserver5 gitlab-runner]# ls -lha
total 12K
drwxr-xr-x 2 root root   76 Dec 29 15:12 .
drwxr-xr-x 3 root root   27 Dec 29 05:31 ..
-rw------- 1 root root  432 Dec 29 05:57 id_ed25519_builder
-rw-r--r-- 1 root root  115 Dec 29 05:57 id_ed25519_builder.pub
[root@pserver5 gitlab-runner]#

I can login to my MAC using this identity file, from the computer where gitlab-runner is running.

[root@pserver5 gitlab-runner]# ssh -i id_ed25519_builder builder@192.168.0.54
Last login: Tue Dec 29 23:30:09 2020 from 192.168.0.10

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
Remote-MacBook-Pro:~ builder$

Here is the .gitlab-ci.yml file for the repository mentioned above:

stages:
  - test

osx-information:
  stage: test
  tags:
    - macos
    - osx
  script:
    - whoami
    - pwd
    - ls -lh
    - uname -a
    - echo "Finally success!" 
  only:
    - master

However, whenever I make any change in any of the files in this repository, the CI/CD pipeline runs, but fails. It keeps saying that it could not find the identity file. I am confused by this, because I can see the identity file and can use it to connect to the target MAC computer correctly. I don't understand what is the problem.

Here are docker logs from the gitlab-runner container:

[root@pserver5 ~]# docker logs -f gitlab-runner_gitlab-runner_1

Checking for jobs... received                       job=936847974 repo_url=https://gitlab.com/kamranazeem/development-on-shared-mac.git runner=hQwrVU_x
WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_rsa: no such file or directory  job=936847974 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936847974 project=23351445 runner=hQwrVU_x
WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_rsa: no such file or directory  job=936847974 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936847974 project=23351445 runner=hQwrVU_x
WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_rsa: no such file or directory  job=936847974 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936847974 project=23351445 runner=hQwrVU_x
ERROR: Job failed (system failure): ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_rsa: no such file or directory  duration=9.032198675s job=936847974 project=23351445 runner=hQwrVU_x
WARNING: Failed to process runner                   builds=0 error=ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_rsa: no such file or directory executor=ssh runner=hQwrVU_x
Configuration loaded                                builds=0
Configuration loaded                                builds=0
Checking for jobs... received                       job=936849815 repo_url=https://gitlab.com/kamranazeem/development-on-shared-mac.git runner=hQwrVU_x
WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_ed25519_builder: no such file or directory  job=936849815 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936849815 project=23351445 runner=hQwrVU_x
WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_ed25519_builder: no such file or directory  job=936849815 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936849815 project=23351445 runner=hQwrVU_x



WARNING: Preparation failed: ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_ed25519_builder: no such file or directory  job=936849815 project=23351445 runner=hQwrVU_x
Will be retried in 3s ...                           job=936849815 project=23351445 runner=hQwrVU_x
ERROR: Job failed (system failure): ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_ed25519_builder: no such file or directory  duration=9.078734741s job=936849815 project=23351445 runner=hQwrVU_x
WARNING: Failed to process runner                   builds=0 error=ssh command Connect() error: getSSHAuthMethods error: open /home/containers-secrets/gitlab-runner/id_ed25519_builder: no such file or directory executor=ssh runner=hQwrVU_x
^C
[root@pserver5 gitlab-runner]#

Thank you for your time.

Edited Dec 30, 2020 by Kamran Azeem