"docker exec -it gitlab update-permissions" does not fix permissions for logrotate configuration files

Summary

update-permissions does not handle logrotate config files which causes instance downtime because logs grow caused no space left.

What is the current bug behavior?

  • the EC2 instance in which gitlab was running (in docker container) had its disk filled up 100% by log files (the largest being api_json.log and production.log...)
  • the logrotate configuration was specified corectly in in gitlab.rb
  • after investigation, it appreared that logrotate was not able to run because of config file permission

The /var/opt/gitlab/logrotate/logrotate.conf (and all others /var/opt/gitlab/logrotate/logrotate.d/*) belongs to a user id 1030 which does not exist inside the container (is it the uid of gitlab user on the host instance)

###into the docker container:

root@99ea1df61266:/# /opt/gitlab/embedded/sbin/logrotate -d -v /var/opt/gitlab/logrotate/logrotate.conf 
WARNING: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

error: Ignoring /var/opt/gitlab/logrotate/logrotate.conf because the file owner is wrong (should be root or user with uid 0).
Reading state from file: /var/lib/logrotate.status
error: error opening state file /var/lib/logrotate.status: No such file or directory
Allocating hash table for state file, size 64 entries

Handling 0 logs

root@99ea1df61266:/# ls -al /var/opt/gitlab/logrotate/logrotate.conf 
-rw-r--r-- 1 1030 1030 587 Jul  6 01:49 /var/opt/gitlab/logrotate/logrotate.conf

root@99ea1df61266:/# getent passwd 1030
<NOTHING>

How to fix the issue

Of course, a manual workaround like this is working :

$ chown -R root: /var/opt/gitlab/logrotate/logrotate.conf
$ chown -R root: /var/opt/gitlab/logrotate/logrotate.d/

$ /opt/gitlab/embedded/sbin/logrotate -d -v /var/opt/gitlab/logrotate/logrotate.conf 
WARNING: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

reading config file /var/opt/gitlab/logrotate/logrotate.conf
including /var/opt/gitlab/logrotate/logrotate.d/nginx
reading config file /var/opt/gitlab/logrotate/logrotate.d/nginx
[...]
Allocating hash table for state file, size 64 entries
Handling 9 logs
[...]

But the ideal approach would be to fix the update-permissions which is, as I understood, supposed to handle these permissions issues.

What is the expected correct behavior?

In our installation process, we launch the docker exec gitlab update-permissions that is supposed to prevent permissions issues between host and container : this script does not seem to handle those logrotate config files. I would expect the update-permissions script to fix the logrotate configuration files permissions in order to avoid this kind of issue.

Relevant logs

Relevant logs 


$ cat /var/log/gitlab/logrotate/current
2021-10-25_01:48:56.51969 error: Ignoring logrotate.conf because the file owner is wrong (should be root or user with uid 0).

$ /opt/gitlab/embedded/sbin/logrotate -d -v /var/opt/gitlab/logrotate/logrotate.conf 
WARNING: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

error: Ignoring /var/opt/gitlab/logrotate/logrotate.conf because the file owner is wrong (should be root or user with uid 0).
Reading state from file: /var/lib/logrotate.status
error: error opening state file /var/lib/logrotate.status: No such file or directory
Allocating hash table for state file, size 64 entries

Handling 0 logs

Details of package version

Provide the package version installation details 
Inside docker container :




root@99ea1df61266:/# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===========================================================
ii  gitlab-ce      14.3.2-ce.0  amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee      <none>       <none>       (no description available)

Environment details

  • Operating System: gitlab docker images on Debian GNU/Linux 10 (buster)
  • Installation Target, remove incorrect values:
    • VM: gitlab image on AWS EC2 instance
  • Is there any other software running on the machine: no (only a docker-compose with gitlab+nginx)
  • Is this a single or multiple node installation? single

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
logrotate['enable'] = true
logging['logrotate_frequency'] = "daily" # rotate logs daily
logging['logrotate_maxsize'] = nil # logs will be rotated when they grow bigger than size specified for `maxsize`, even before the specified time interval (daily, weekly, monthly, or yearly)
logging['logrotate_size'] = nil # do not rotate by size by default
logging['logrotate_rotate'] = 30 # keep 30 rotated logs
logging['logrotate_compress'] = "compress" # see 'man logrotate'
logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
logging['logrotate_postrotate'] = nil # no postrotate command by default
logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz
Edited by Guillaume Dupin