docs: Document lockfile write timing and fix the backend check

What does this MR do?

Fixes two documentation defects on the container registry metadata database page. Both were found while answering a self-managed customer's question about upgrading a Geo deployment to 19.0, and both were reproduced on a lab.

Prefer mode never says when a lockfile is written

The "Prefer mode" section documents what the registry does for each of the four lockfile states, and that part is accurate. What it never states is when a lockfile comes into existence. Every branch of the decision is phrased as a property of the storage, as though the marker were simply a fact about a bucket.

The registry writes the marker at startup, when it starts against storage that already contains metadata; the metadata import also writes one when it completes. A push never writes it.

That omission makes the third branch unreachable by reasoning:

Neither lockfile exists: The registry is a fresh installation. It requires a configured and reachable database to start and does not fall back to legacy storage.

"The registry is a fresh installation" is an inference, and it is false for any registry that has not restarted since its storage was first populated. That includes an ordinary single-node install: the registry starts against an empty bucket, images are pushed to it, and no marker is written, because a push never writes the marker.

Observed on a single node, 19.2.1, registry v4.40.2-gitlab. A registry started against empty storage, then pushed to:

Objects in storage Lockfiles
After the push, registry still running 8 none
After one gitlab-ctl restart registry, nothing pushed 9 filesystem-in-use

The ninth object is the lockfile. Nothing was pushed between the two reads, which rules out the push path writing it.

A Geo secondary site is the same mechanism with a longer window rather than a separate case: Geo replicates images by pushing them through the secondary's own registry API (the mechanism is read from source; the lockfile consequence was measured), so its storage also fills while its registry is already running. Confirmed on a two-site Geo pair, both sites holding a complete 8-object catalog with no lockfile on either.

I mention the Geo case only because it is where the window stays open longest and where an operator is most likely to meet it during an upgrade. The defect is not specific to Geo and the fix does not depend on it.

What happens when the marker is missing

"Requires a configured and reachable database to start" reads as a precondition. It is an outcome, and there are two, depending on whether the configured database is reachable.

Reachable, with the schema, and empty. The registry starts, adopts the empty database, stamps database-in-use, and serves an empty catalog over full storage with a 200 OK. Measured on 19.2.1 against a database holding the registry schema at 20 tables and zero repository rows, with storage holding 12 objects of real content, lockfiles stripped and prefer mode set explicitly. On a Geo secondary this is indistinguishable from a correct new site still awaiting replication.

Not usable. The registry logs using the metadata database, never falls back, and does not start. gitlab-ctl reconfigure fails at registry_database_migrations[registry] and the service crash-loops over a complete and healthy filesystem catalog.

"Reachable" is not the test that matters, and the trigger deserves precision. Omnibus decides whether to attempt migrations with pg_isready, which reports only that the server is accepting connections; it does not authenticate and does not check the database exists (read from the omnibus recipes; the failure it produces was measured). So a registry PostgreSQL that is listening but rejects the configured credentials takes this branch too. I hit exactly that on 19.2.1 with SASL auth: FATAL: password authentication failed: the reconfigure failed at the registry database migration while that site's registry kept serving, because its lockfile was present and it had fallen back to legacy metadata. A site that has never used a registry database and has no credentials configured is the realistic shape for meeting this during an upgrade.

Recovery differs between the two, and the quiet one is worse here as well. In the loud case, restoring enabled: false and restarting brought the catalog straight back. In the quiet case the registry has already adopted the empty database and stamped database-in-use, so enabled: false afterwards lands in the fourth row of the decision table below and the registry refuses to start until that lockfile is removed by hand. Nothing in storage is damaged either way. But an operator meeting either one inside an upgrade window has no way to connect it to the prefer mode section, which is what this MR fixes.

One asymmetry belongs on the page plainly: a site with a working registry database gets the silent version, and a site without one gets the loud version. The operator who prepared most thoroughly is the one most likely to get the failure that does not announce itself.

The decision table

Rather than leave the six combinations as prose, the MR adds a table. Every row was measured individually on 19.2.1 against storage holding a complete catalog:

Lockfile in storage enabled Registry starts Catalog served
None false Yes, uses legacy metadata and writes filesystem-in-use Complete
filesystem-in-use 'prefer' Yes, falls back to legacy metadata Complete
None 'prefer' Yes, adopts the database and writes database-in-use Empty, with 200 OK
database-in-use false No, registry metadata database in use None
Both 'prefer' No, database-in-use and filesystem-in-use lockfiles present None
filesystem-in-use false Yes, uses legacy metadata Complete

Exactly one row is silent. The two that refuse to start both name the reason and link a troubleshooting page, which is good behavior and worth making visible next to the one that does not.

This MR adds the write timing, corrects the inference in the third branch, adds the table, and adds a pre-upgrade lockfile check with the one-restart remedy to "Existing installations".

The last row needs a caveat, which the table itself creates

The final row, filesystem-in-use with the database disabled, serves the complete catalog. That is what I measured and it is what an operator reverting a never-migrated site will see.

It is not true for a site that has already served a push from the database. "Before you start" on this page already says so:

After you enable the database, you must continue to use it. [...] disabling it after this point causes the registry to lose visibility on all images written to it while the database was active.

Confirmed on 19.2.1, and the numbers make the sentence concrete. A site migrated to the database, pushed to three more times, then returned to storage metadata by flipping the lockfile back by hand, which isolates what the storage alone serves (a configuration-only revert refuses to start, per the decision table's fourth row):

Tags
Registry metadata database 6
Filesystem metadata in storage 3
Served after reverting 3

Storage object count did not change, so no blobs were lost; the blobs for the missing three are still present with no filesystem metadata referring to them. The registry starts cleanly and logs no error.

Adding a table that presents that combination as benign, without noting the condition, would make this page slightly worse. So a caveat follows the table and links to the page's revert documentation, which carries the warning and the supported procedure.

The backend verification command cannot succeed as written

"Check the registry API response header" documents:

curl --silent --head "https://registry.example.com/v2/" | grep --ignore-case gitlab-container-registry-database-enabled

The request is unauthenticated. Against a GitLab container registry, /v2/ answers 401 Unauthorized and the response carries none of the gitlab-container-registry-* headers, so the grep matches nothing, prints nothing, and exits 1.

Observed side by side on 19.2.1:

Status Header Documented pipeline output
Unauthenticated, as documented 401 absent nothing, exit 1
With a token 200 false the header

Silence is not a neutral result here. The procedure asks a yes-or-no question and tells the reader the answer arrives as true or false, so an empty result reads as "not true" at exactly the moment an operator is deciding whether a migration completed or whether a site is safe to upgrade. The two other methods on the same page, the lockfiles and the startup log line, both work as documented; this one is listed first and disagrees with them.

This MR adds authentication and names the third outcome explicitly.

A note on terminology

The page already carries several names for this: "filesystem metadata", "legacy metadata storage", "legacy storage", and "legacy filesystem storage". New prose does not coin another and does not try to impose a convention retroactively, so the one line that would have named the data directly says "the metadata in storage" instead. Where the new text needs a short form it uses "legacy metadata", the wording the registry's own fallback log line already uses on this page. The literal lockfile names stay as filesystem-in-use and database-in-use, in code spans.

The page's existing "filesystem" uses are untouched. Normalizing them page-wide would be a separate change from this one.

!249943 is the companion Geo documentation MR against administration/geo/replication/container_registry.md and .../disaster_recovery/planned_failover.md; its registry preflight check links to the verification section this MR corrects. Different pages and different groups, so they are separate, but worth reviewing together.

!246587 is an open draft that restructures the earlier part of this page, extracting a tutorial and replacing the "Before you start" list. It does not touch the prefer mode, decision table, or verification sections this MR changes, and the two do not overlap textually.

Author's checklist

  • Follows the documentation style guide
  • markdownlint-cli2 clean (0 errors, using the version CI pins, 0.22.1)
  • Vale re-measured under the rules CI actually enforces (no exclusions): 26 warnings on origin/master and 26 on this branch, the identical set, so zero net change
  • Editing an existing page only, so no new product availability details block is required
  • Behavioral claims executed on a lab; the two source-read mechanisms (the pg_isready gate in the omnibus recipes, Geo replicating through the secondary's own registry API) are named as such in the body
  • Request review from the Container Registry technical writer

AI-Generated Content Disclosure: This MR was prepared with assistance from Claude Code. The output has been reviewed for correctness, verified on a lab against GitLab 19.2.1 with registry v4.40.2-gitlab, and validated against the documentation style guide.

Edited by Andrew Dunn

Merge request reports

Loading