gitlab-ee omnibus gitaly tls options don't get written into gitaly.toml on reconfigure
(ubuntu 18.04) gitlab-ee 11.7.5-ee.0 title says it all, gitlab-ee omnibus gitaly tls settings, eg:
# /etc/gitlab/gitlab.rb
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/ssl/gitaly-selfsigned.crt"
gitaly['key_path'] = "/etc/ssl/private/gitaly-selfsigned.key"are not placed into /var/opt/gitlab/gitaly/config.toml with # gitlab-ctl reconfigure,
the box I'm working with was provisioned from-scratch yesterday, with no services installed besides gitlab-ee omnibus, and an example configuration used from https://docs.gitlab.com/ee/administration/gitaly/#on-gitaly-server-nodes
relevant parts of the gitlab.rb conf
# /etc/gitlab/gitlab.rb
gitaly['listen_addr'] = "0.0.0.0:8075"
gitaly['auth_token'] = '********'
gitaly['storage'] = [
{ 'name' => 'default', 'path' => '/srv/gitlab/default/repositories' },
{ 'name' => 'storage1', 'path' => '/srv/gitlab/storage1/repositories' },
]
# To use TLS for Gitaly you need to add
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/ssl/gitaly-selfsigned.crt"
gitaly['key_path'] = "/etc/ssl/private/gitaly-selfsigned.key"relevant parts of the reconfigure log
# gitlab-ctl reconfigure
* template[Create Gitaly config.toml] action create
- update content in file /var/opt/gitlab/gitaly/config.toml from e6**** to 98****
--- /var/opt/gitlab/gitaly/config.toml 2019-02-12 19:44:06.609155882 +0000
+++ /var/opt/gitlab/gitaly/.chef-config20190212-20553-ygkzez.toml 2019-02-12 19:45:19.702151948 +0000
@@ -7,18 +7,24 @@
socket_path = '/var/opt/gitlab/gitaly/gitaly.socket'
bin_dir = '/opt/gitlab/embedded/bin'
+# Optional: listen on a TCP socket. This is insecure (no authentication)
+listen_addr = '0.0.0.0:8075'
# Optional: export metrics via Prometheus
prometheus_listen_addr = 'localhost:9236'
[[storage]]
name = 'default'
-path = '/var/opt/gitlab/git-data/repositories'
+path = '/srv/gitlab/default/repositories'
+[[storage]]
+name = 'storage1'
+path = '/srv/gitlab/storage1/repositories'
[logging]
[auth]
+token = '********'
[gitaly-ruby]
dir = "/opt/gitlab/embedded/service/gitaly-ruby" and lastly, relevant parts of the gitaly log, unsuprisingly due to the config lines not being written, it doesn't bind to the tls port
# /var/log/gitlab/gitaly/current
level=info msg="Starting Gitaly" version="Gitaly, version 1.12.2, built 20190205.235246"
level=warning msg="git path not configured. Using default path resolution" resolvedPath=/opt/gitlab/embedded/bin/git
level=warning msg="git path not configured. Using default path resolution" resolvedPath=/opt/gitlab/embedded/bin/git
level=info msg="finished tempdir cleaner walk" storage=default time_ms=0
level=info msg="listening on unix socket" address=/var/opt/gitlab/gitaly/gitaly.socket
level=info msg="listening at tcp address" address="0.0.0.0:8075"
level=info msg="Starting prometheus listener" address="localhost:9236"
level=info msg="finished tempdir cleaner walk" storage=storage1 time_ms=0Follow up, I've manually added the missing lines to the config.toml (an act I understand I can't rely on in production with the provided/managed isv software)
relevant toml diff:
--- /var/opt/gitlab/gitaly/config.toml.orig 2019-02-12 20:01:54.369321224 +0000
+++ /var/opt/gitlab/gitaly/config.toml 2019-02-12 19:52:28.419349254 +0000
@@ -9,11 +9,16 @@
# Optional: listen on a TCP socket. This is insecure (no authentication)
listen_addr = '0.0.0.0:8075'
+tls_listen_addr = '0.0.0.0:9999'
# Optional: export metrics via Prometheus
prometheus_listen_addr = 'localhost:9236'
+[tls]
+certificate_path = '/etc/ssl/gitaly-selfsigned.crt'
+key_path = '/etc/ssl/private/gitaly-selfsigned.key'
+
[[storage]]
name = 'default'
path = '/srv/gitlab/default/repositories'pseudo-secondary issue,
if I add the missing config lines, gitaly doesn't start as root to read the private keys then setuid/setgid; I understand that as a vendor you may not be able to offer that system-integrated behaviour; so I've worked around this by storing the private keys without filesystem protection.
# /var/log/gitlab/gitaly/current
level=fatal msg="open /etc/ssl/private/gitaly-selfsigned.key: permission denied"corrected by storing the ssl data somewhere else (for me /opt/gitaly-ssl/, owned by git:git)
with the gitaly tls config manually installed gitaly starts the tls socket, the gitaly log does not indicate that it bound to the tls socket, but the network stats show the socket is bound:
# ss -nlpt | grep 9999
LISTEN 0 1024 *:9999 *:* users:(("gitaly",pid=23065,fd=6))to summarize, it seems like the issue is in the omnibus gitaly config.toml template, which appears to mismatch current administration documentation.
Thank you for your time. I hope I've provided enough detail about the topic