Standalone Gitaly is not UDP forwarding its logs
<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
In a scalable context, when Gitaly is running alone on its VM, installed with Omnibus, UDP log forwarding is not working.
### Steps to reproduce
Install Gitaly as standalone.
Configure it to forward UDP logs
### Example Project
N/A
### What is the current *bug* behavior?
Logs are not forwarded
### What is the expected *correct* behavior?
Logs are forwarded
### Relevant logs and/or screenshots
* We have separated Gitaly, PostgreSQL, Redis and Rails+Shell+Sidekiq+Puma... All standard Omnibus checks are failing because of that, ths checks are not adapted to scalable architecture.
* The Rails+Shell+Sidekiq+Puma component is correctly sending its logs
* Sent logs can be seen by this command: `tcpdump -i eth0 port 514`
* The Gitaly component is not sending anything
* But we have rsyslog fowarding on this VM for other logs (ssh access for instance) and these logs are sent without any issue
* It looks like we have to activate something more that what is in the documentation here: https://about.gitlab.com/blog/2014/12/08/ship-log-data-off-site-using-udp/
* Here is the configuration file used for Gitaly, as Jinja2 template. Log configuration is at the end:
```
# From https://docs.gitlab.com/ce/administration/gitaly/#3-gitaly-server-configuration
# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
# Enable prometheus monitoring - comment out if you disable monitoring services above.
# This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port.
prometheus['listen_address'] = '0.0.0.0:9090'
# Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load
# balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://{{ gitlab_public_host }}/'
# Authentication token to ensure only authorized servers can communicate with
# Gitaly server
gitaly['auth_token'] = '{{ gitaly_token }}'
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['listen_addr'] = "0.0.0.0:8075"
# Local Git data directories
# We get the value in the storages variable (default value in role should not be changed without knowing what is done here)
git_data_dirs({ '{{ storages[component_instance | int] | default("NONE") }}' => { 'path' => '/var/opt/gitlab/git-data' } })
# Logging format
gitaly['logging_format'] = 'default'
##! remote host to ship log messages to via UDP
logging['udp_log_shipping_host'] = "{{ graylog_host }}"
```
### Output of checks
#### Results of GitLab environment info
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
The result is coherent with the configuration and scalable architecture.
```
System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)
```
</pre>
</details>
#### Results of GitLab application Check
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
The result is coherent with the configuration and scalable architecture.
```
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... FAIL: 14:failed to connect to all addresses
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
```
</pre>
</details>
### Possible fixes
¯\\_(ツ)_/¯
issue