Skip to content

Periodically truncate log files

Mitchell Nielsen requested to merge truncate-logs into master

Summary

Truncates log files periodically:

  • Disabled by default
  • Configurable truncation interval
  • Configurable maximum file size

Closes https://gitlab.com/gitlab-org/charts/components/gitlab-logger/-/issues/9

Testing

docker-compose

To test, run the command from the README and confirm that it exits 0:

docker-compose -f docker-compose.truncate-logs.yml build && docker-compose -f docker-compose.truncate-logs.yml up

CI

CNG and downstream Charts pipeline: https://gitlab.com/gitlab-org/build/CNG/-/pipelines/691535187 ✅

Charts installation

global:
  gitlabVersion: test-logger-truncate
  extraEnv:
    GITLAB_LOGGER_DEBUG: true
    GITLAB_LOGGER_TRUNCATE_LOGS: true
    GITLAB_LOGGER_TRUNCATE_INTERVAL: 10
    GITLAB_LOGGER_MAX_FILESIZE: 10

gitlab:
  gitlab-exporter:
    image:
      tag: test-logger-truncate
  gitaly:
    image:
      tag: test-logger-truncate
  gitlab-shell:
    image:
      tag: test-logger-truncate

Periodically checking size of logs, you can see them grow and then get trimmed down to 0 bytes:

git@gitlab-webservice-default-57dc54488f-vltj8:/$ cd /var/log/gitlab

git@gitlab-webservice-default-57dc54488f-vltj8:/var/log/gitlab$ ls -lah
total 16K
drwxr-xr-x 2 git  git  4.0K Nov 10 20:04 .
drwxr-xr-x 4 root root 4.0K Nov 10 20:02 ..
-rw-r--r-- 1 git  git     0 Nov 10 20:08 .gitkeep
-rw-r--r-- 1 git  git     0 Nov 10 20:08 api_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 application.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 application_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 auth_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 database_load_balancing.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 grpc.log
-rw-r--r-- 1 git  git   442 Nov 10 20:08 production.log
-rw-r--r-- 1 git  git  2.1K Nov 10 20:08 production_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 puma.stderr.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 puma.stdout.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 service_measurement.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 sidekiq_client.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 web_exporter.log

git@gitlab-webservice-default-57dc54488f-vltj8:/var/log/gitlab$ ls -lah
total 8.0K
drwxr-xr-x 2 git  git  4.0K Nov 10 20:04 .
drwxr-xr-x 4 root root 4.0K Nov 10 20:02 ..
-rw-r--r-- 1 git  git     0 Nov 10 20:08 .gitkeep
-rw-r--r-- 1 git  git     0 Nov 10 20:08 api_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 application.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 application_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 auth_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 database_load_balancing.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 grpc.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 production.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 production_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 puma.stderr.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 puma.stdout.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 service_measurement.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 sidekiq_client.log
-rw-r--r-- 1 git  git     0 Nov 10 20:08 web_exporter.log

Same with Gitaly:

git@gitlab-gitaly-0:/$ ls -la /var/log/gitaly
total 12
drwxr-xr-x 2 git  root 4096 Nov 10 20:50 .
drwxr-xr-x 1 root root 4096 Oct 25 03:42 ..
-rw-r--r-- 1 git  git     0 Nov 10 20:51 gitaly.log
-rw-r--r-- 1 git  git   516 Nov 10 20:51 gitaly_ruby_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:51 gitlab-shell.log

git@gitlab-gitaly-0:/$ ls -la /var/log/gitaly
total 8
drwxr-xr-x 2 git  root 4096 Nov 10 20:50 .
drwxr-xr-x 1 root root 4096 Oct 25 03:42 ..
-rw-r--r-- 1 git  git     0 Nov 10 20:51 gitaly.log
-rw-r--r-- 1 git  git     0 Nov 10 20:51 gitaly_ruby_json.log
-rw-r--r-- 1 git  git     0 Nov 10 20:51 gitlab-shell.log

References

Some references used during research for posterity:

Edited by Mitchell Nielsen

Merge request reports