[Spike] Address ephemeral storage consumption due to log file size

Summary

Gitlab Pods become evicted because of their ephemeral-storage usage.

Like Omnibus GitLab the Helm chart gitlab needs a built-in logrotate service.

Steps to reproduce

Let Gitlab chart running for months or simulate a big log file with load tests.

Configuration used

gitlab:
  task-runner:
   ....
  gitaly:
    persistence:
      ....
    resources:
      ....
    logging:
      level: info

(fyi) logging.level: info is the default value

Current behavior

Gitlab Pods become evicted because of their ephemeral-storage usage, exemple with

The node was low on resource: ephemeral-storage. Container gitlab-gitaly-0 was using XXXXXXKi, which exceeds its request of XXX.

Same may happen on every pods using ephemeral-storage to store his logs. (gitlab-shell, gitlab-webservice)

Expected behavior

The logs rotate using new settings passed in the chart values.yaml.

Versions

• Chart: 4.0.5
• Platform:
  • Cloud: EKS
• Kubernetes: (kubectl version)
  • Client: v1.17.2
  • Server: v1.16.13-eks-2ba888
• Helm: (helm version)
  • Client: v2.16.9
  • Server: v2.16.9

Relevant logs

[root@ip-192-168-0-100 _data]# pwd
/var/lib/docker/volumes/63fccaf24c0d735863dfgdff48ghjhgjk5d40546045a1e0d714cdaabe6/_data
[root@ip-192-168-0-100 _data]# ls -lh
total 8.0G
-rw-r--r-- 1 ec2-user ec2-user 8.3K Sep 24 12:51 gitaly_hooks.log
-rw-r--r-- 1 ec2-user root     8.0G Sep 25 10:21 gitaly.log
-rw-r--r-- 1 ec2-user root      11M Sep 25 10:21 gitlab-shell.log