garbage-collect refuses to run with "not compatible with database metadata" when the registry has fallen back to filesystem metadata in prefer mode

Summary

When database.enabled is "prefer" and the registry has fallen back to legacy filesystem metadata at startup, garbage-collect still refuses to run and reports that it is incompatible with database metadata. In that state the registry is demonstrably not using the database, offline GC is the only mechanism that can reclaim anything, and the error text points the operator at online GC, which provably has nothing to do.

The result is a registry with no working garbage collection at all, and an error message that actively directs the operator away from the cause.

Conditions

  1. database.enabled: "prefer" (the default for new installations since 19.0).
  2. A filesystem-in-use lockfile is present in the configured registry storage, which is the expected state for any registry that has served legacy metadata and has not been imported.
  3. The registry therefore falls back at startup and logs:
level=warning msg="database prefer mode enabled, but found filesystem metadata: falling back to legacy metadata"
level=info msg="registry filesystem metadata in use"

Running offline GC then fails:

$ registry garbage-collect /var/opt/gitlab/registry/config.yml -m -d
Error: the garbage-collect command is not compatible with database metadata, please use online garbage collection instead. database.enabled now defaults to "prefer". To use filesystem metadata, set database.enabled to false in your configuration

Meanwhile online GC has nothing queued, because nothing is being written to the metadata database:

$ gitlab-ctl registry-database gc-stats
=== Blob Review Queue ===
Tasks Pending Removal: 0
Long Overdue Tasks: 0
High Retry Tasks: 0
=== Manifest Review Queue ===
Tasks Pending Removal: 0
Long Overdue Tasks: 0
High Retry Tasks: 0

Root cause

Verified at v4.41.0-gitlab.

GCCmd resolves the configuration from the config file and then gates on config.Database.IsEnabled(), at registry/root.go#L198-200.

IsEnabled() returns false for prefer only once PreferFallback is true:

func (d Database) IsEnabled() bool {
	return d.Enabled != DatabaseEnabledUnset && d.Enabled != DatabaseEnabledFalse && !d.PreferFallback
}

PreferFallback is set only inside the serving application's startup path, at registry/handlers/app.go#L2465, within the fallback branch at app.go#L2457-2468.

The garbage-collect CLI never constructs an App and never evaluates the lockfiles, so PreferFallback is always false in that process. prefer therefore always evaluates as enabled, and the command cannot distinguish "the registry is using the database" from "the registry is configured for prefer and has fallen back to filesystem metadata".

The command is reporting on configuration, not on the backend the running registry actually selected.

Impact

For a registry in prefer-fallback:

  • Online GC cannot reclaim anything, because no metadata is written to the database.
  • Offline GC refuses to run.
  • Cleanup policies still delete tag references, but nothing reclaims the underlying layers.

Storage therefore grows without bound. In one self-managed case this ran for roughly two months and reached 482 GB before the operator found the startup warning; the second and third suggestions in the error message were both dead ends, and the message gave no hint that the database was not in use. The same diagnosis has come up in several support tickets.

Note also that the remediation the message suggests, setting database.enabled to false, is the correct action for a registry that is genuinely on filesystem metadata. An operator who does not realise they are already in fallback has no reason to believe that applies to them.

Proposal

Have garbage-collect determine the effective backend the way the application does, rather than trusting configuration alone:

  • Check the lockfiles in the configured storage before refusing.
  • If filesystem-in-use is present and enabled is prefer, the registry is on filesystem metadata, so allow the command to proceed.
  • If database-in-use is present, keep refusing, and name the signal the decision was based on.

If reading storage from the CLI is undesirable, a smaller fix still helps materially: reword the message to distinguish the configured value from the effective backend, and point at the lockfiles and the startup log line as the way to tell which is in use.

References