Segmentation fault when the Gitlab runner starts a job.
Summary
With the latest GitLab version from master, installed with the GDK and a local runner, when the runner picks up a job, GitLab encounters a segmentation fault and the runner fails to run the job.
Steps to reproduce
- Install GitLab with the GDK.
- Install a runner and configure it (docker or Ubuntu package encounter the issue, probably any kind).
- Start a pipeline.
Example Project
Can't give an example project as the problem doesn't occur on Gitlab.com
What is the current bug behavior?
-
The
gdk run
log shows a segmentation fault. -
The runner complains about an HTTP error code 502.
-
The job output windows stays black, no log appears there.
What is the expected correct behavior?
The runner picks up the job and runs it
Relevant logs and/or screenshots
gdk run log:
15:16:15 rails-web.1 | /home/groulot/src/gdk-ee/gitlab/app/services/ci/register_job_service.rb:48: [BUG] Segmentation fault at 0x0000000000000004
15:16:15 rails-web.1 | ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
15:16:15 rails-web.1 |
15:16:15 rails-web.1 | -- Control frame information -----------------------------------------------
15:16:15 rails-web.1 | c:0128 p:0047 s:0803 E:002228 BLOCK /home/groulot/src/gdk-ee/gitlab/app/services/ci/register_job_service.rb:48 [FINISH]
15:16:15 rails-web.1 | c:0127 p:---- s:0799 e:000798 IFUNC
15:16:15 rails-web.1 | c:0126 p:---- s:0796 e:000795 CFUNC :each
15:16:15 rails-web.1 | c:0125 p:0018 s:0792 E:001d70 METHOD /home/groulot/.rvm/gems/ruby-2.5.3/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38 [FINISH]
...
15:16:16 rails-web.1 | [NOTE]
15:16:16 rails-web.1 | You may have encountered a bug in the Ruby interpreter or extension libraries.
15:16:16 rails-web.1 | Bug reports are welcome.
15:16:16 rails-web.1 | For details: http://www.ruby-lang.org/bugreport.html
Gitlab-runner log:
nov. 20 15:34:58 gabriel gitlab-runner[20607]: WARNING: Checking for jobs... failed runner=d1d4dc46 status=502 Bad Gateway
Output of checks
Results of GitLab environment info
(output of bundle exec rake gitlab:env:info RAILS_ENV=development
)
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Proxy: no Current User: groulot Using RVM: yes RVM Version: 1.29.3 Ruby Version: 2.5.3p105 Gem Version: 2.7.7 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.0.6 Git Version: 2.19.1 Sidekiq Version:5.2.3 Go Version: go1.9.4 linux/amd64
GitLab information Version: 11.5.0-pre Revision: 27a8297dd61 Directory: /home/groulot/src/gdk-ee/gitlab DB Adapter: postgresql DB Version: 11.0 URL: http://localhost:3000 HTTP Clone URL: http://localhost:3000/some-group/some-project.git SSH Clone URL: ssh://groulot@localhost:2222/some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 8.4.2 Repository storage paths:
- default: /home/groulot/src/gdk-ee/repositories Hooks: /home/groulot/src/gdk-ee/gitlab-shell/hooks Git: /usr/bin/git
Results of GitLab application Check
(output of bundle exec rake gitlab:check RAILS_ENV=development SANITIZE=true
)
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 8.4.1 ? ... OK (8.4.2) Running /home/groulot/src/gdk-ee/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /home/groulot/src/gdk-ee/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... no Trying to fix error automatically. ...Success Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... no Try fixing it: sudo chmod 700 /home/groulot/src/gdk-ee/gitlab/public/uploads For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 2/1 ... yes 2/2 ... yes 3/3 ... yes 4/4 ... yes 5/5 ... yes 6/6 ... yes 7/7 ... yes 8/8 ... yes 57/9 ... yes 57/10 ... yes 57/11 ... yes 57/12 ... yes 57/13 ... yes 2/18 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.9.5 ? ... yes (2.19.1) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1542725605 sudo mv /home/groulot/.ssh/id_rsa ~/gitlab-check-backup-1542725605 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 49 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Possible fixes
in