gitlab-ctl reconfigure stuck in recursive loop on startup on Docker for Windows
Summary
Using the latest gitlab-ce Docker image with mounted volumes or folder binds, the initial Chef configuration gets stuck in an unending (recursive?)loop eventually throwing a stack level too deep error. This does not happen if NO volumes/binds are mounted. I'm using Docker for Windows with the gitlab-ce:latest image.
Steps to reproduce
-
using CLI:
The CLI command that I use to fire up my container from PowerShell or a .ps1 script is as follows:
docker run --hostname --publish 4430:443 --publish 8880:80 --publish 2224:22 --name ${containerName} --restart always --volume ${pwd}/gitlab/config:/etc/gitlab --volume ${pwd}/gitlab/logs:/var/log/gitlab --volume ${pwd}/gitlab/opt:/var/opt gitlab/gitlab-ce:latestThe $containerName variable is defined in the script beforehand. I bind my host folder to /var/opt instead of /var/opt/gitlab due to another shell permissions error
-
using docker-compose:
web: image: 'gitlab/gitlab-ce:latest' restart: always hostname: 'gitlab.example.com' environment: GITLAB_OMNIBUS_CONFIG: | external_url 'https://gitlab.example.com' # Add any other gitlab.rb configuration here, each on its own line ports: - '8080:80' - '443:443' - '22:22' volumes: - '.\srv\gitlab\config:/etc/gitlab' - '.\srv\gitlab\logs:/var/log/gitlab' - '.\srv\gitlab\opt:/var/opt'I just picked up the docker-compose configuration from the GitLab Docker tutorial website, changed the host part of the volume mount specs to match the Windows convention + use the current directory as the the base and changed the port mappings
What is the current bug behavior?
gitlab-ctl reconfigure gets stuck in an infinite loop ending up with a stack error when volumes or binds are attached to the container on Docker Windows. Please see logs below
What is the expected correct behavior?
gitlab-ctl reconfigure should complete when volumes are attached just like it does when volumes are NOT attached
Something like this: (log contents when the container is run without any volumes or folder binds)
Recipe: redis::enable
* runit_service[redis] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create
- create new directory /opt/gitlab/sv/redis
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* template[/opt/gitlab/sv/redis/run] action create
- create new file /opt/gitlab/sv/redis/run
- update content in file /opt/gitlab/sv/redis/run from none to da365d
--- /opt/gitlab/sv/redis/run 2019-07-20 10:44:11.347110400 +0000
+++ /opt/gitlab/sv/redis/.chef-run20190720-33-1vke0sq 2019-07-20 10:44:11.347110400 +0000
@@ -1 +1,6 @@
+#!/bin/sh
+exec 2>&1
+
+umask 077
+exec chpst -P -U gitlab-redis:gitlab-redis -u gitlab-redis:gitlab-redis /opt/gitlab/embedded/bin/redis-server /var/opt/gitlab/redis/redis.conf
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/redis/log] action create
- create new directory /opt/gitlab/sv/redis/log
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/redis/log/main] action create
- create new directory /opt/gitlab/sv/redis/log/main
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* template[/opt/gitlab/sv/redis/log/run] action create
- create new file /opt/gitlab/sv/redis/log/run
- update content in file /opt/gitlab/sv/redis/log/run from none to af1017
--- /opt/gitlab/sv/redis/log/run 2019-07-20 10:44:11.407110400 +0000
+++ /opt/gitlab/sv/redis/log/.chef-run20190720-33-43igha 2019-07-20 10:44:11.407110400 +0000
@@ -1 +1,3 @@
+#!/bin/sh
+exec svlogd -tt /var/log/gitlab/redis
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* template[/var/log/gitlab/redis/config] action create
- create new file /var/log/gitlab/redis/config
- update content in file /var/log/gitlab/redis/config from none to 623c00
--- /var/log/gitlab/redis/config 2019-07-20 10:44:11.427110400 +0000
+++ /var/log/gitlab/redis/.chef-config20190720-33-1uw62f0 2019-07-20 10:44:11.417110400 +0000
@@ -1 +1,7 @@
+s209715200
+n30
+t86400
+!gzip
+
+
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/redis/env] action create
- create new directory /opt/gitlab/sv/redis/env
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create
- create new directory /opt/gitlab/sv/redis/control
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* link[/opt/gitlab/init/redis] action create
- create symlink at /opt/gitlab/init/redis to /opt/gitlab/embedded/bin/sv
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* ruby_block[restart_service] action run (skipped due to only_if)
* ruby_block[restart_log_service] action create
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create (up to date)
* directory[/opt/gitlab/sv/redis/env] action create (up to date)
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/redis] action create
- create symlink at /opt/gitlab/service/redis to /opt/gitlab/sv/redis
* ruby_block[wait for redis service socket] action run
- execute the ruby block wait for redis service socket
- execute the ruby block restart_log_service
* ruby_block[reload_log_service] action create
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create (up to date)
* directory[/opt/gitlab/sv/redis/env] action create (up to date)
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/redis] action create (up to date)
* ruby_block[wait for redis service socket] action run (skipped due to not_if)
- execute the ruby block reload_log_service
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/redis] action create (up to date)
* ruby_block[wait for redis service socket] action run (skipped due to not_if)Relevant logs and/or screenshots
* ruby_block[restart_log_service] action create
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create
- change mode from '0755' to '0644'
* directory[/opt/gitlab/sv/redis/env] action create (up to date)
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* ruby_block[reload_log_service] action create
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create
- change mode from '0755' to '0644'
* directory[/opt/gitlab/sv/redis/env] action create (up to date)
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* ruby_block[reload_log_service] action createHow the recursive calling eventually ends: Please scroll to the extreme right in the frame below to view the contents
=========================================================================
=======
Error executing action `enable` on resource 'runit_service[redis]'
=========================================================================
=======
Chef::Exceptions::MultipleFailures
----------------------------------
Multiple failures occurred:
* SystemStackError occurred in delayed notification: stack level too deep
* SystemStackError occurred in delayed notification: stack level too deep
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_r
unit_service.rb:242:in `block in <class:RunitService>'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/red
is_service.rb
65: runit_service 'redis' do
66: down node['redis']['ha']
67: template_name 'redis'
68: options({
69: service: 'redis',
70: log_directory: redis_log_dir
71: }.merge(params))
72: log_options node['gitlab']['logging'].to_hash.merge(node['redis'
].to_hash)
73: end
74:
75: if node['gitlab']['bootstrap']['enable']
76: execute "/opt/gitlab/bin/gitlab-ctl start redis" do
77: retries 20
78: end
79: end
80: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/defini
tions/redis_service.rb:65:in `block in from_file'
runit_service("redis") do
params {:socket_group=>"git", :name=>nil}
provider Chef::Provider::RunitService
action :enable
updated true
updated_by_last_action true
default_guard_interpreter :default
service_name "redis"
enabled false
running false
masked nil
options {:service=>"redis", :log_directory=>"/var/log/gitlab/redis", :s
ocket_group=>"git", :name=>nil}
pattern "redis"
start_command "start"
stop_command "stop"
status_command "status"
restart_command "restart"
supports {:restart=>true, :reload=>true, :status=>true}
sv_bin "/opt/gitlab/embedded/bin/sv"
sv_dir "/opt/gitlab/sv"
service_dir "/opt/gitlab/service"
lsb_init_dir "/opt/gitlab/init"
log true
owner "root"
group "root"
restart_on_update true
run_template_name "redis"
log_template_name "redis"
check_script_template_name "redis"
finish_script_template_name "redis"
sv_templates true
log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeou
t"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=
>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>true, "ha"=>false, "hz"=>10, "dir"=>"/var/opt/gitlab/redis", "log_directory"=>"/var/log/gitlab/redis", "username"=>"gitlab-redis", "group"=>"gitlab-redis"
, "uid"=>nil, "gid"=>nil, "shell"=>"/bin/false", "home"=>"/var/opt/gitlab/redis", "bind"=>"127.0.0.1", "port"=>0, "maxclients"=>"10000", "maxmemory"=>"0", "maxmemory_policy"=>"noeviction", "maxmemory_samples"=>5, "tcp_backlog"=>511, "tcp_timeout"=>60, "tcp_keepalive"=>30
0, "password"=>nil, "unixsocket"=>"/var/opt/gitlab/redis/redis.socket", "unixsocketperm"=>"777", "master"=>true, "master_name"=>"gitlab-redis", "master_ip"=>nil, "master_port"=>6379, "master_password"=>nil, "client_output_buffer_limit_normal"=>"0 0 0", "client_output_buf
fer_limit_slave"=>"256mb 64mb 60", "client_output_buffer_limit_pubsub"=>"32mb 8mb 60", "save"=>["900 1", "300 10", "60 10000"], "announce_ip"=>nil, "announce_port"=>nil}
declared_type :runit_service
cookbook_name "redis"
recipe_name "enable"
service_mirror # Declared in
service("redis") do
provider Chef::Provider::Service::Simple
action [:nothing]
default_guard_interpreter :default
service_name "redis"
enabled nil
running nil
masked nil
pattern "redis"
start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/
embedded/bin/sv start /opt/gitlab/service/redis"
stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/e
mbedded/bin/sv stop /opt/gitlab/service/redis"
status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab
/embedded/bin/sv status /opt/gitlab/service/redis"
restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitla
b/embedded/bin/sv restart /opt/gitlab/service/redis"
supports {:restart=>true, :reload=>true, :status=>true}
end
end
System Info:
------------
chef_version=13.6.4
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run (skipped due to not_if)
Running handlers:
Running handlers complete
Chef Client failed. 1014 resources updated in 03 minutes 23 seconds
There was an error running gitlab-ctl reconfigure:
Multiple failures occurred:
* SystemStackError occurred in delayed notification: stack level too deep
* SystemStackError occurred in delayed notification: stack level too deep
PS C:\devops\piworks-teamcity>I couldn't find any issue similar to mine while going list of issues. Hopefully it's something minor I'm missing that won't prevent me from being able to use GitLab in my environment.
Docker info:
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git commit: 6247962
Built: Sun Feb 10 04:12:31 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: trueEdited by Umair Ahmed