Fix registry DB migration log accumulation

What does this MR do?

Fix registry DB migration log accumulation

The registry database migration resource writes its output to a per-invocation timestamped file (db-migrations-YYYY-MM-DD-HH-MM-SS.log), and the resource has no idempotency guard, so it runs unconditionally on every reconfigure. Logrotate is also not configured for the registry log directory. Long-lived instances accumulate one file per reconfigure indefinitely; any process tailing /var/log/gitlab/registry/*.log (notably gitlab-ctl tail, which uses tail --follow=name --retry) opens one file descriptor per file, eventually exhausting limits and surfacing as "Too many open files".

Append migration output to a single rolling db-migrations.log with a timestamped run header per invocation, and add the registry service to the default logrotate services list so the standard rotation policy (daily, 30 rotations, gzip, copytruncate) applies to the new file.

Functional validation

Rebased 2026-09-17 onto master, which now invokes /opt/gitlab/embedded/bin/registry database migrate up [-s] <registry_dir>/config.yml directly (see f0eab058). The rolling-log change is layered on top of that command; the run header reads registry database migrate up, with -s appended when SKIP_POST_DEPLOYMENT_MIGRATIONS=true.

Validated with the image built by this MR's downstream pipeline (gitlab-ee:19.4.0-rfbranch.2857938825.a09f1d47-0, arm64) using embedded PostgreSQL and registry['database']['enabled'] = true.

Before the fix (gitlab-ee:18.11.0-ee.0, 2026-05-20) — 5 reconfigures produce 5 files:

-rw------- registry registry 11713 db-migrations-2026-05-20-20-55-08.log
-rw------- registry registry   242 db-migrations-2026-05-20-20-55-37.log
-rw------- registry registry   242 db-migrations-2026-05-20-20-55-46.log
-rw------- registry registry   242 db-migrations-2026-05-20-20-55-56.log
-rw------- registry registry   242 db-migrations-2026-05-20-20-56-06.log

After the fix — 6 reconfigures (initial install + 5) produce a single rolling file with one header per run:

-rw------- registry registry 12587 db-migrations.log

=== 2026-09-17T12:41:00Z registry database migrate up ===
pre-deployment:
20210503145024_create_top_level_namespaces_table
...
=== 2026-09-17T12:42:31Z registry database migrate up ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.032s
=== 2026-09-17T12:42:40Z registry database migrate up ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.028s
...

The find -L … | xargs tail --follow=name --retry pipeline that gitlab-ctl tail uses now holds file descriptors for one file only.

gitlab-ctl reconfigure renders /var/opt/gitlab/logrotate/logrotate.d/registry (/var/log/gitlab/registry/*.log, su registry registry, daily, rotate 30, compress, copytruncate). A forced rotation followed by another reconfigure with SKIP_POST_DEPLOYMENT_MIGRATIONS=true:

-rw------- registry registry  175 db-migrations.log
-rw------- registry registry 2635 db-migrations.log.1.gz

=== 2026-09-17T12:43:39Z registry database migrate up -s ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.010s

Upgrade and rollback behaviour

  • Upgrade: the first reconfigure after upgrading starts writing db-migrations.log and renders the registry logrotate rule. Legacy db-migrations-<timestamp>.log files already on disk are not removed. The new *.log rule rotates each of them once into a .1.gz and leaves an empty .log behind (notifempty then skips them), and gitlab-ctl tail still opens each empty .log. Instances that have already hit the descriptor limit should remove the legacy files manually after upgrading:

    rm -f /var/log/gitlab/registry/db-migrations-[0-9]*.log /var/log/gitlab/registry/db-migrations-[0-9]*.log.*.gz
  • Rollback: the previous cookbook resumes writing timestamped files. db-migrations.log remains as a single static file, and the stale logrotate.d/registry entry is no longer included by the regenerated logrotate.conf, so it is inert.

Closes #9874 (closed)

Related to #4342 (the registry log accumulation was split out of that older issue into 9874)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.
  • The UBT version and corresponding checksum hash have been updated and referenced in the merge request if applicable.
    • UBT EE pipeline (Trigger:ee-package-ubt) is green

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by João Alexandre Cunha

Merge request reports

Loading
Loading