Fix registry DB migration log accumulation
What does this MR do?
Fix registry DB migration log accumulation
The registry database migration resource writes its output to
a per-invocation timestamped file (db-migrations-YYYY-MM-DD-HH-MM-SS.log),
and the resource has no idempotency guard, so it runs unconditionally
on every reconfigure. Logrotate is also not configured for the registry
log directory. Long-lived instances accumulate one file per reconfigure
indefinitely; any process tailing /var/log/gitlab/registry/*.log
(notably gitlab-ctl tail, which uses tail --follow=name --retry)
opens one file descriptor per file, eventually exhausting limits and
surfacing as "Too many open files".
Append migration output to a single rolling db-migrations.log with a
timestamped run header per invocation, and add the registry service to
the default logrotate services list so the standard rotation policy
(daily, 30 rotations, gzip, copytruncate) applies to the new file.
Functional validation
Rebased 2026-09-17 onto master, which now invokes
/opt/gitlab/embedded/bin/registry database migrate up [-s] <registry_dir>/config.yml
directly (see f0eab058). The rolling-log change is layered on top of that
command; the run header reads registry database migrate up, with -s
appended when SKIP_POST_DEPLOYMENT_MIGRATIONS=true.
Validated with the image built by this MR's downstream pipeline
(gitlab-ee:19.4.0-rfbranch.2857938825.a09f1d47-0, arm64) using embedded
PostgreSQL and registry['database']['enabled'] = true.
Before the fix (gitlab-ee:18.11.0-ee.0, 2026-05-20) — 5 reconfigures produce 5 files:
-rw------- registry registry 11713 db-migrations-2026-05-20-20-55-08.log
-rw------- registry registry 242 db-migrations-2026-05-20-20-55-37.log
-rw------- registry registry 242 db-migrations-2026-05-20-20-55-46.log
-rw------- registry registry 242 db-migrations-2026-05-20-20-55-56.log
-rw------- registry registry 242 db-migrations-2026-05-20-20-56-06.logAfter the fix — 6 reconfigures (initial install + 5) produce a single rolling file with one header per run:
-rw------- registry registry 12587 db-migrations.log
=== 2026-09-17T12:41:00Z registry database migrate up ===
pre-deployment:
20210503145024_create_top_level_namespaces_table
...
=== 2026-09-17T12:42:31Z registry database migrate up ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.032s
=== 2026-09-17T12:42:40Z registry database migrate up ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.028s
...The find -L … | xargs tail --follow=name --retry pipeline that gitlab-ctl tail
uses now holds file descriptors for one file only.
gitlab-ctl reconfigure renders /var/opt/gitlab/logrotate/logrotate.d/registry
(/var/log/gitlab/registry/*.log, su registry registry, daily, rotate 30,
compress, copytruncate). A forced rotation followed by another reconfigure with
SKIP_POST_DEPLOYMENT_MIGRATIONS=true:
-rw------- registry registry 175 db-migrations.log
-rw------- registry registry 2635 db-migrations.log.1.gz
=== 2026-09-17T12:43:39Z registry database migrate up -s ===
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.010sUpgrade and rollback behaviour
-
Upgrade: the first reconfigure after upgrading starts writing
db-migrations.logand renders the registry logrotate rule. Legacydb-migrations-<timestamp>.logfiles already on disk are not removed. The new*.logrule rotates each of them once into a.1.gzand leaves an empty.logbehind (notifemptythen skips them), andgitlab-ctl tailstill opens each empty.log. Instances that have already hit the descriptor limit should remove the legacy files manually after upgrading:rm -f /var/log/gitlab/registry/db-migrations-[0-9]*.log /var/log/gitlab/registry/db-migrations-[0-9]*.log.*.gz -
Rollback: the previous cookbook resumes writing timestamped files.
db-migrations.logremains as a single static file, and the stalelogrotate.d/registryentry is no longer included by the regeneratedlogrotate.conf, so it is inert.
Related issues
Closes #9874 (closed)
Related to #4342 (the registry log accumulation was split out of that older issue into 9874)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
- MR title and description are up to date, accurate, and descriptive.
- MR targeting the appropriate branch.
- Latest Merge Result pipeline is green.
- When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.
- The UBT version and corresponding checksum hash have been updated and referenced in the merge request if applicable.
- UBT EE pipeline (
Trigger:ee-package-ubt) is green
- UBT EE pipeline (
For GitLab team members
If you don't have access to this, the reviewer should trigger these jobs for you during the review process.
- The manual
Trigger:ee-packagejobs have a green pipeline running against latest commit.- To debug QA failures, refer to the Investigate QA failures section.
- If
config/softwareorconfig/patchesdirectories are changed, make sure thebuild-package-on-all-osjob within theTrigger:ee-packagedownstream pipeline succeeded. - If you are changing anything SSL related, then the
Trigger:package:fipsmanual job within theTrigger:ee-packagedownstream pipeline must succeed. - If CI configuration is changed, the branch must be pushed to
dev.gitlab.orgto confirm regular branch builds aren't broken.
Expected (please provide an explanation if not completing)
- Test plan indicating conditions for success has been posted and passes.
- Documentation created/updated.
- Tests added.
- Integration tests added to GitLab QA.
- Equivalent MR/issue for the GitLab Chart opened.
- Validate potential values for new configuration settings. Formats such as integer
10, duration10s, URIscheme://user:passwd@host:portmay require quotation or other special handling when rendered in a template and written to a configuration file.