How to debug gitlab runner
I noticed strange behavior of my gitlab runner. It stop for some time (30 min or even one hour), and than back to work. It had happend 3 times during 3 months, but in every of this case I didn't know why and how to debug it.
Today again !
My environment:
Gitlab Host: GitLab CE 11.0.3
Gitlab Runner Host:
Version: 9.5.1
Git revision: 96b34ccf
Git branch: 9-5-stable
GO version: go1.8.3
Built: Wed, 04 Oct 2017 16:26:27 +0000
OS/Arch: linux/amd64
When it happend all jobs was pending.
First off all I stopped all jobs from admin. Developers still was running jobs, and pending queue grows.
I switched to gitlab runner machine:
# gitlab-ci-multi-runner verify
Running in system-mode.
ERROR: Verifying runner... is removed runner=7fe40d51
ERROR: Verifying runner... is removed runner=59a7aa2f
ERROR: Verifying runner... is removed runner=2c60a715
ERROR: Verifying runner... is removed runner=3d3e4bc0
ERROR: Verifying runner... is removed runner=8053e381
FATAL: Failed to verify runners
Ok something wrong with runners, but why ?
/var/log/messages - nothing
# gitlab-ci-multi-runner restart
Sep 3 13:22:56 gitlabrunner1 gitlab-runner[24895]: time="2018-09-03T13:22:56+02:00" level=info msg="Starting multi-runner from /etc/gitlab-runner/config.toml ..." builds=0 #012<nil>
Sep 3 13:22:56 gitlabrunner1 gitlab-runner[24895]: time="2018-09-03T13:22:56+02:00" level=info msg="Running in system-mode." #012<nil>
Sep 3 13:22:56 gitlabrunner1 gitlab-runner[24895]: time="2018-09-03T13:22:56+02:00" level=info #012<nil>
Sep 3 13:22:56 gitlabrunner1 gitlab-runner[24895]: time="2018-09-03T13:22:56+02:00" level=info msg="Configuration loaded" builds=0 #012<nil>
Sep 3 13:22:56 gitlabrunner1 gitlab-runner[24895]: time="2018-09-03T13:22:56+02:00" level=info msg="Metrics server disabled" #012<nil>
Only this ...
And after some time, everything goes on.
What more can I do, to diagnose the reason ?