Skip to content

Segmentation fault when the Gitlab runner starts a job.

Summary

With the latest GitLab version from master, installed with the GDK and a local runner, when the runner picks up a job, GitLab encounters a segmentation fault and the runner fails to run the job.

Steps to reproduce

  • Install GitLab with the GDK.
  • Install a runner and configure it (docker or Ubuntu package encounter the issue, probably any kind).
  • Start a pipeline.

Example Project

Can't give an example project as the problem doesn't occur on Gitlab.com

What is the current bug behavior?

  • The gdk run log shows a segmentation fault.

  • The runner complains about an HTTP error code 502.

  • The job output windows stays black, no log appears there.

What is the expected correct behavior?

The runner picks up the job and runs it

Relevant logs and/or screenshots

gdk run log:

15:16:15 rails-web.1              | /home/groulot/src/gdk-ee/gitlab/app/services/ci/register_job_service.rb:48: [BUG] Segmentation fault at 0x0000000000000004
15:16:15 rails-web.1              | ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
15:16:15 rails-web.1              | 
15:16:15 rails-web.1              | -- Control frame information -----------------------------------------------
15:16:15 rails-web.1              | c:0128 p:0047 s:0803 E:002228 BLOCK  /home/groulot/src/gdk-ee/gitlab/app/services/ci/register_job_service.rb:48 [FINISH]
15:16:15 rails-web.1              | c:0127 p:---- s:0799 e:000798 IFUNC 
15:16:15 rails-web.1              | c:0126 p:---- s:0796 e:000795 CFUNC  :each
15:16:15 rails-web.1              | c:0125 p:0018 s:0792 E:001d70 METHOD /home/groulot/.rvm/gems/ruby-2.5.3/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38 [FINISH]
...
15:16:16 rails-web.1              | [NOTE]
15:16:16 rails-web.1              | You may have encountered a bug in the Ruby interpreter or extension libraries.
15:16:16 rails-web.1              | Bug reports are welcome.
15:16:16 rails-web.1              | For details: http://www.ruby-lang.org/bugreport.html

Gitlab-runner log:

nov. 20 15:34:58 gabriel gitlab-runner[20607]: WARNING: Checking for jobs... failed                runner=d1d4dc46 status=502 Bad Gateway

Output of checks

Results of GitLab environment info

(output of bundle exec rake gitlab:env:info RAILS_ENV=development)

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 16.04
Proxy:		no
Current User:	groulot
Using RVM:	yes
RVM Version:	1.29.3
Ruby Version:	2.5.3p105
Gem Version:	2.7.7
Bundler Version:1.16.2
Rake Version:	12.3.1
Redis Version:	3.0.6
Git Version:	2.19.1
Sidekiq Version:5.2.3
Go Version:	go1.9.4 linux/amd64


GitLab information
Version:	11.5.0-pre
Revision:	27a8297dd61
Directory:	/home/groulot/src/gdk-ee/gitlab
DB Adapter:	postgresql
DB Version:	11.0
URL:		http://localhost:3000
HTTP Clone URL:	http://localhost:3000/some-group/some-project.git
SSH Clone URL:	ssh://groulot@localhost:2222/some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	8.4.2
Repository storage paths:
  • default: /home/groulot/src/gdk-ee/repositories Hooks: /home/groulot/src/gdk-ee/gitlab-shell/hooks Git: /usr/bin/git

Results of GitLab application Check

(output of bundle exec rake gitlab:check RAILS_ENV=development SANITIZE=true)

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 8.4.1 ? ... OK (8.4.2)
Running /home/groulot/src/gdk-ee/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /home/groulot/src/gdk-ee/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... no
Trying to fix error automatically. ...Success
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... no
Try fixing it:
sudo chmod 700 /home/groulot/src/gdk-ee/gitlab/public/uploads
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
3/3 ... yes
4/4 ... yes
5/5 ... yes
6/6 ... yes
7/7 ... yes
8/8 ... yes
57/9 ... yes
57/10 ... yes
57/11 ... yes
57/12 ... yes
57/13 ... yes
2/18 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.9.5 ? ... yes (2.19.1)
Git user has default SSH configuration? ... no
Try fixing it:
mkdir ~/gitlab-check-backup-1542725605
sudo mv /home/groulot/.ssh/id_rsa ~/gitlab-check-backup-1542725605
For more information see:
doc/ssh/README.md in section "SSH on the GitLab server"
Please fix the error above and rerun the checks.
Active users: ... 49
Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)


Checking GitLab ... Finished

Possible fixes

in

Edited by Gilbert Roulot