logrotate defaults: set notifempty and some additional documentation on logging behaviour

What does this MR do?

This change to logrotate will prevent empty logs being rotated, and reduce the number of empty log files that get created on Omnibus installations.

The documentation revision includes some more details on Omnibus logs. The rotation of runit logs is not intuitive to many GitLab administrators, and it'd be useful to Support if we had information about the precise behavior of these logs in the documentation.

---

Specific benefits of the logrotate change:

1. The database migration logs get created once, and made unique with a datestamp. Rotating these
   • Is untidy. Fourteen DB migrate logs will increase to 420 after thirty days; this setting limits the increase to 28.
   • Results in key logs being unavailable for troubleshooting once the files have been rotated 30 times. Support encounters this often. Plus, customers have to fish the 'real' logs out from all the 20 bytes empty compressed logs.

-rw-------. 1 git git   20 Dec  3 00:00 gitlab-rails-db-migrate-2019-11-18-15-07-22.log.10.gz
-rw-------. 1 git git   20 Dec  3 00:00 gitlab-rails-db-migrate-2019-10-08-03-35-21.log.10.gz
-rw-------. 1 git git   20 Dec  4 00:00 gitlab-rails-db-migrate-2019-11-18-15-07-22.log.9.gz
-rw-------. 1 git git   20 Dec  4 00:00 gitlab-rails-db-migrate-2019-10-08-03-35-21.log.9.gz
-rw-------. 1 git git 6378 Dec  4 10:20 gitlab-rails-db-migrate-2019-12-04-10-19-51.log.9.gz
-rw-------. 1 git git   20 Dec  5 00:31 gitlab-rails-db-migrate-2019-12-04-10-19-51.log.8.gz
-rw-------. 1 git git   20 Dec  5 00:31 gitlab-rails-db-migrate-2019-11-18-15-07-22.log.8.gz
-rw-------. 1 git git   20 Dec  5 00:31 gitlab-rails-db-migrate-2019-10-08-03-35-21.log.8.gz
-rw-------. 1 git git   20 Dec  6 00:31 gitlab-rails-db-migrate-2019-10-08-03-35-21.log.7.gz
-rw-------. 1 git git   20 Dec  6 00:31 gitlab-rails-db-migrate-2019-12-04-10-19-51.log.7.gz
-rw-------. 1 git git   20 Dec  6 00:31 gitlab-rails-db-migrate-2019-11-18-15-07-22.log.7.gz

2. Other Omnibus logs also get rotated even if they're empty. Again, this creates a lot of volume in certain directories; such as nginx, where on an installation without all the functionality in use (or infrequently used) the logs get rotated regardless.

-rw-r--r--.  1 root root            20 Dec  8 00:31 error.log.16.gz
-rw-r--r--.  1 root root            20 Dec  8 00:31 access.log.16.gz
-rw-r--r--.  1 root root         93905 Dec  9 00:31 gitlab_access.log.16.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 gitlab_registry_error.log.15.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 gitlab_registry_access.log.15.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 gitlab_pages_error.log.15.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 gitlab_error.log.15.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 error.log.15.gz
-rw-r--r--.  1 root root            20 Dec  9 00:31 access.log.15.gz
-rw-r--r--.  1 root root           100 Dec  9 14:48 gitlab_pages_access.log.15.gz
-rw-r--r--.  1 root root         96358 Dec 10 00:31 gitlab_access.log.15.gz
-rw-r--r--.  1 root root            20 Dec 10 00:31 error.log.14.gz
-rw-r--r--.  1 root root            20 Dec 10 00:31 access.log.14.gz

IF a log is written to infrequently, this change will keep 30 copies (using the default setting) not 30 days' of logs.

There are a number of other ways this could be addressed, with varying levels of complexity, some of which are suggested in the comments of #3054

Scope: the majority of log directories use runit (so they have a log called current) and tai64n (.s) format files for rotation.

Related issues

• #3054
• gitlab#12540 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on the dev.gitlab.org mirror for this branch if the change is touching anything besides documentation or internal cookbooks
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened

Test Planning

What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing?

Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance.

• Unit test changes
• Integration test changes
• End-to-end test change

See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning -->

• The notifempty setting has been tested for several months on an Omnibus 12.x server to check for any side effects.
• The main risk from not having effective log rotation is that the filesystem containing /var/log/gitlab fills up.
• The effect of this setting is to suspend log rotation when a file is empty so log rotation should continue to be effective for the logs that matter, ie: those with any data in them.