Standalone Gitaly is not UDP forwarding its logs

Summary

In a scalable context, when Gitaly is running alone on its VM, installed with Omnibus, UDP log forwarding is not working.

Steps to reproduce

Install Gitaly as standalone. Configure it to forward UDP logs

Example Project

N/A

What is the current bug behavior?

Logs are not forwarded

What is the expected correct behavior?

Logs are forwarded

Relevant logs and/or screenshots

• We have separated Gitaly, PostgreSQL, Redis and Rails+Shell+Sidekiq+Puma... All standard Omnibus checks are failing because of that, ths checks are not adapted to scalable architecture.
• The Rails+Shell+Sidekiq+Puma component is correctly sending its logs
• Sent logs can be seen by this command: tcpdump -i eth0 port 514
• The Gitaly component is not sending anything
• But we have rsyslog fowarding on this VM for other logs (ssh access for instance) and these logs are sent without any issue
• It looks like we have to activate something more that what is in the documentation here: https://about.gitlab.com/blog/2014/12/08/ship-log-data-off-site-using-udp/
• Here is the configuration file used for Gitaly, as Jinja2 template. Log configuration is at the end:

# From https://docs.gitlab.com/ce/administration/gitaly/#3-gitaly-server-configuration

# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false

# Enable prometheus monitoring - comment out if you disable monitoring services above.
# This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port.
prometheus['listen_address'] = '0.0.0.0:9090'

# Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false

# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load
# balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://{{ gitlab_public_host }}/'

# Authentication token to ensure only authorized servers can communicate with
# Gitaly server
gitaly['auth_token'] = '{{ gitaly_token }}'

# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['listen_addr'] = "0.0.0.0:8075"

# Local Git data directories
# We get the value in the storages variable (default value in role should not be changed without knowing what is done here)
git_data_dirs({ '{{ storages[component_instance | int] | default("NONE") }}' => { 'path' => '/var/opt/gitlab/git-data' } })

# Logging format
gitaly['logging_format'] = 'default'

##! remote host to ship log messages to via UDP
logging['udp_log_shipping_host'] = "{{ graylog_host }}"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

The result is coherent with the configuration and scalable architecture.
```
System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.5p114
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.24.1
Sidekiq Version:5.2.7
Go Version:	unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:48:in `block (3 levels) in '
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)
```

Results of GitLab application Check

Expand for output related to the GitLab application check

The result is coherent with the configuration and scalable architecture.

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... FAIL: 14:failed to connect to all addresses

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... Exception: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... Exception: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... Exception: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

Is authorized keys file accessible? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

¯\_(ツ)_/¯