Two-site Geo archetype with the container registry on the metadata database

What this adds

A two-site Geo archetype for the simulation skill: gce/geo.sh plus a just geo recipe. The existing single-VM bases cannot reach the class of question that only appears once a second site exists, which is what replicates, what a secondary serves while it is still backfilling, and what survives a promotion.

geo.sh up [--registry-mode legacy|database] [--version VER]
geo.sh license [path]          # PRIMARY only
geo.sh status | verify | failover | down
geo.sh registry-migrate <primary|secondary>

Two design choices are load-bearing:

• Object storage is per site. Each VM runs its own MinIO, so a failover test is not reading the storage of the site that just died.
• The secondary's registry database is standalone, not the omnibus one. replicate-geo-database makes every database in the cluster read-only, and a registry must write its metadata, so on a secondary it has to point at a PostgreSQL that Geo is not replicating.

Registry backend is a per-site flag, which makes the asymmetric mid-cutover window reachable rather than theoretical.

Why it is shaped this way

Every fix below was found by running it, not by reading it. Each one produced a cluster that reported success and did nothing, which is the failure mode that manufactures false findings.

Symptom	Cause
Both nodes inert, geo status lists two healthy sites	Site records named primary/secondary never match GeoNode.current_node_name, which falls back to the node's own URL, so current_node is nil on both
Predicates keep answering nil after the records are fixed	Gitlab::Geo caches current_node; needs expire_cache!
replicate reports complete, pg_is_in_recovery false	gitlab-ctl stop times out on sidekiq and aborts the tool; the detached heredoc ended in echo, masking rc=1
Blob pulls fail NoSuchKey against a bucket of the same name	regionendpoint was 127.0.0.1, so the other site resolved the redirect to its own empty storage
The license key is invalid on a valid license	scp preserved the file's 0600 mode; gitlab-rails runner runs as git
registry database migrate up targets the wrong database	Legacy mode rendered no password; 18.3+ preprovisions a registry database and role inside omnibus PostgreSQL, which is the one Geo replicates read-only
Reconfigure always reports success	Same trailing-echo mask, third instance

Three of those are the same masking trap in different functions, so the exit status is now propagated in replicate, promote and reconfigure alike.

Guards that refuse rather than report

The recurring hazard here is a lab that looks built and replicates nothing, because its output is then indistinguishable from a genuine finding. Three checks now fail closed:

• geo_require_license refuses when Geo is unlicensed, because the sync service is a silent no-op without a license: it returns without raising and prints a synced path while nothing moves.
• geo_require_bound refuses unless each node recognises itself.
• prep-secondary asserts pg_is_in_recovery. This caught a real failure on its first run.

Validation

Run end to end on an 18.11.7 pair:

• Gitlab::Geo.license_allows? true, both sites bound.
• Container registry replication observed: matching digest on both sides, secondary tag list converged, full metadata tree in its bucket.
• Event-driven replication observed, unforced. A tag pushed at 20:50:26Z was picked up by the secondary's log cursor at 20:50:27.698Z and the sync finished at 20:50:28.195Z, with the secondary bucket going from 24 objects to 32. That path is what 8eee559 adds. Before it this script configured no notifications at all, so every earlier measurement in this branch was of the forced path and only looked like replication working on its own.
• Failover observed: primary? flips true, pg_is_in_recovery false, and the registry keeps the same pid with an unchanged config and lockfile, confirming promotion contains no registry step.
• Mixed-backend window observed: a legacy secondary replicated a new tag, carrying a genuinely new layer, from a primary already on the metadata database.

Not covered

• MinIO stands in for other S3 implementations.
• One small repository and two tags, so nothing here is a timing figure. The event latency above is a single observation on an idle pair.
• Deletion events and notification retry. Only the push path has been watched.
• registry-migrate runs the import against a live read-write registry rather than a read-only one. Safe in a lab where nothing is pushing, and it should not be lifted into an operational procedure as-is.

Also in this branch

One unrelated line in gce/aigw.md, rewording "the exact test or probe that will guard the fix" to "that guards the fix". The vale gate flags future tense on that line whether or not this branch touches it, so it is cleared here rather than left to fail the next unrelated MR.