consul service postgresql_service failing on db host - no access to /opt/gitlab/embedded/node
Summary
On provisioning HA setup using 10.5.3 consul service /opt/gitlab/bin/gitlab-ctl repmgr-check-master
returns succssfully but consul log shows
2018-03-08_05:37:02.00803 2018/03/08 05:37:02 [WARN] agent: Check 'service:postgresql' is now warning
The issue is directory/file permissions on /opt/gitlab/embedded/node
preventing user gitlab-consul access
Steps to reproduce
- Using 10.5.3
- Provision 3 x consul node cluster
- Provision 1 x db and set as master
What is the current bug behavior?
Consul check succeeds but postgresql_service is not synced.
What is the expected correct behavior?
Consul postgresql_service to be synced.
Relevant logs
Relevant logs
10.5 root@db1:~# ls -ld /opt/gitlab/embedded/nodes /opt/gitlab/embedded/nodes/* drwx------ 2 root root 4096 Mar 8 01:14 /opt/gitlab/embedded/nodes -rw------- 1 root root 212195 Mar 8 01:15 /opt/gitlab/embedded/nodes/db1.vagrant.jsonroot@db1:
# chmod og+rx /opt/gitlab/embedded/nodes root@db1:# chmod go+r /opt/gitlab/embedded/nodes/* root@db1:~# ls -ld /opt/gitlab/embedded/nodes /opt/gitlab/embedded/nodes/* drwxr-xr-x 2 root root 4096 Mar 8 01:14 /opt/gitlab/embedded/nodes -rw-r--r-- 1 root root 212195 Mar 8 01:15 /opt/gitlab/embedded/nodes/db1.vagrant.json
Details of package version
Provide the package version installation details
10.5.3 on all nodes
Environment details
- Operating System: Ubuntu 16.04
- Installation Target, remove incorrect values:
- VM: Vagrant
- Installation Type, remove incorrect values:
- New Installation
- Is there any other software running on the machine: No
- Is this a single or multiple node installation? Multi
- Resources
- Consul CPU: 2
- Consul Memory total: 256MB
- DB CPU: 2
- DB Memory total: 1024MB
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url "http://app1.vagrant" roles ['postgres_role'] postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' postgresql['shared_preload_libraries'] = 'repmgr_funcs' gitlab_rails['auto_migrate'] = false consul['services'] = %w(postgresql) postgresql['pgbouncer_user_password'] = '*********************' postgresql['sql_user_password'] = '*********************' postgresql['max_wal_senders'] = 3 postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) consul['configuration'] = { bind_addr: "10.6.0.21", retry_join: %w(consul1.vagrant consul2.vagrant consul3.vagrant) }
Workarounds
Updated permissions on /opt/gitlab/embedded/nodes to be the same as 10.4
chmod go+rx /opt/gitlab/embedded/nodes chmod go+r /opt/gitlab/embedded/nodes/* # ls -ld /opt/gitlab/embedded/nodes /opt/gitlab/embedded/nodes/* drwxr-xr-x 2 root root 4096 Mar 8 01:14 /opt/gitlab/embedded/nodes -rw-r--r-- 1 root root 212195 Mar 8 01:15 /opt/gitlab/embedded/nodes/db1.vagrant.json
This workaround is only good until the next gitlab-ctl reconfigure
which will overwrite those changes.