After update 10.5.2: Segmentation fault at 0x0000000000000000 when doing backups and in unicorn logs
Summary
After updating GitLab from 10.4.4 to 10.5.2 doing backups errors with segmentation faults on different steps. The problem persists after updating from 10.5.2 to 10.5.3, running gitlab-ctl reconfigure and after a reboot of the OpenVZ VPS.
With every execution of sudo gitlab-rake gitlab:backup:create
the segfault occurs later in the process until after the ~ 5th execution the backup is finished successfully.
Besides that we sporadically encounter errors when browsing the GitLab instance (viewing commits, creating projects etc.) which results in segfaults in the unicorn log at /var/log/gitlab/unicorn/unicorn_stderr.log
Steps to reproduce
- running
sudo gitlab-rake gitlab:backup:create
or - browsing the GitLab web interface (it appears randomly and never happens again after a refresh)
What is the current bug behavior?
- segfault when doing backups: https://gist.github.com/adriankirchner/ccaf3265f6086f7027b5ee9c846dc924
- segfault from the unicorn error log: https://gist.github.com/adriankirchner/8a742295e55ead12ccb90bd0f62d30e7
What is the expected correct behavior?
- finishing backup jobs
- don't see GitLab errors while browsing the GitLab instance
Relevant logs
Relevant logs
- segfault while doing backups: https://gist.github.com/adriankirchner/ccaf3265f6086f7027b5ee9c846dc924 - segfault from the unicorn error log: https://gist.github.com/adriankirchner/8a742295e55ead12ccb90bd0f62d30e7
Details of package version
Provide the package version installation details
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-===================================================================================================-======================================================-======================================================-============================================================================================================================================================================================================ ii gitlab-ce 10.5.3-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis) un gitlab-ee (no description available)
Environment details
- Operating System:
Ubuntu 16.04.4 LTS
- Installation Target, remove incorrect values:
- Other:
OpenVZ VPS (Kernel 2.6.32-042stab127.2)
- Other:
- Installation Type, remove incorrect values:
- Upgrade from version
10.4.4
- Upgrade from version
- Is there any other software running on the machine: no
- Is this a single or multiple node installation?
single node
- Resources
- CPU:
8 cores Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
- Memory total:
16gb total, 11gb available
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab.mydomain.sc'gitlab_rails['time_zone'] = 'Europe/Brussels' gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.sc' gitlab_rails['artifacts_enabled'] = true
gitlab_rails['backup_path'] = "/var/opt/gitlab/backups" gitlab_rails['backup_keep_time'] = 604800
gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "smtp.mandrillapp.com" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = "XXX" gitlab_rails['smtp_password'] = "XXXXXXX" gitlab_rails['smtp_domain'] = "example.com" gitlab_rails['smtp_authentication'] = "login" gitlab_ci['smtp_enable'] = true gitlab_ci['smtp_address'] = "smtp.mandrillapp.com" gitlab_ci['smtp_port'] = 587 gitlab_ci['smtp_user_name'] = "XXX" gitlab_ci['smtp_password'] = "XXXXXXXX" gitlab_ci['smtp_domain'] = "example.com" gitlab_ci['smtp_authentication'] = "login"
nginx['redirect_http_to_https'] = true nginx['redirect_http_to_https_port'] = 80 nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain.sc.crt" nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mydomain.sc.key" nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2" # recommended by https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/ nginx['custom_gitlab_server_config'] = "location ^~/.well-known/acme-challenge/ { proxy_pass http://xx.xx.xx.xx; }"
logging['logrotate_rotate'] = 7 # keep 30 rotated logs
ci_external_url 'http://ci.mydomain.sc'