Gitlab CI Pipelines stuck pending regards of stage state
Summary
Gitlab CI Pipelines stuck pending regardless of whether or not they have passed of failed
Steps to reproduce
git push or any kind of commit which kicks off the build process on any
What is the current bug behavior?
Pipelines do not progress.
What is the expected correct behavior?
Should progress to next stage or mark failed
Relevant logs and/or screenshots
Examples here fail but other stages from other repos pass but do not progress to next stage
Here is a dump of a Runner job failing, there appears to be no unusual errors
Appending trace to coordinator... ok code=202 job=2584 job-log=0-1552656 job-status=running runner=b568ea69 sent-log=1552656-1552656 status=202 Accepted
Appending trace to coordinator... ok code=202 job=2584 job-log=0-1552683 job-status=running runner=b568ea69 sent-log=1552656-1552683 status=202 Accepted
Container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e finished with exit code 1 job=2584 project=1499 runner=b568ea69
Executing on /runner-b568ea69-project-1499-concurrent-0-build the set -eo pipefail
set +o noclobber
: | eval ''
exit 0
job=2584 project=1499 runner=b568ea69
Attaching to container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e ... job=2584 project=1499 runner=b568ea69
Starting container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e ... job=2584 project=1499 runner=b568ea69
Waiting for attach to finish 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e ... job=2584 project=1499 runner=b568ea69
Waiting for container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e ... job=2584 project=1499 runner=b568ea69
Appending trace to coordinator... ok code=202 job=2584 job-log=0-1556868 job-status=running runner=b568ea69 sent-log=1552683-1556868 status=202 Accepted
Container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e finished with <nil> job=2584 project=1499 runner=b568ea69
WARNING: Job failed: exit code 1 job=2584 project=1499 runner=b568ea69
Submitting job to coordinator... ok job=2584 runner=b568ea69
Removed container 4becb1cd5b0e2e6719ecace290bf5331d3150d6c140d7203b168066585ae937e with <nil> job=2584 project=1499 runner=b568ea69
Removed container c716c4f6aa82342edc30af6d3d7baf4357456abee6119f113c97e4ba405f707d with <nil> job=2584 project=1499 runner=b568ea69
Checking for jobs... nothing runner=b568ea69
Feeding runners to channel builds=0
Checking for jobs... nothing runner=b568ea69
Feeding runners to channel builds=0Output of checks
Gitlab-CE 9.4.5 running on-prem
Results of GitLab environment info
gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.13.4
Sidekiq Version:5.0.0
Go Version: unknown
GitLab information
Version: 9.4.5
Revision: 140292e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://gitlab.insight-centre.org
HTTP Clone URL: https://gitlab.insight-centre.org/some-group/some-project.git
SSH Clone URL: git@gitlab.insight-centre.org:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 5.3.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true) Checking GitLab Shell ...GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ...
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ...
Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.4) Active users: ... 377
Checking GitLab ... Finished
Additional Info
I've checked some of the sidekiq runtime parameters to ensure they are as expected
/opt/gitlab/embedded/service/gitlab-rails/config/sidekiq_queues.yml
/opt/gitlab/embedded/service/gitlab-rails/config/sidekiq_queues.yml
and also the launch of sidekiq
/opt/gitlab/embedded/bin/bundle exec sidekiq \
-C /opt/gitlab/embedded/service/gitlab-rails/config/sidekiq_queues.yml \
-e production \
-r /opt/gitlab/embedded/service/gitlab-rails \
-t 4 \
-c 25I've also enabled this in gitlab.rb with no apparent effect
gitlab_rails['stuck_ci_jobs_worker_cron'] = "0 0 * * *"