Geo: Skip duplicate rows when bulk inserting registries
What does this MR do and why?
Geo::BaseRegistry.insert_for_model_ids used bulk_insert! with the default on_duplicate: :raise. On registries with a unique index on the model foreign key, a race between Geo::RegistryConsistencyWorker and the sync path's find_or_initialize_by raised ActiveRecord::RecordNotUnique and aborted the whole batch (up to 10,000 rows). Because the batcher advances its cursor before the insert runs, the failed range was skipped and those records stayed unregistered and unsynced until the batcher wrapped around the whole table.
This MR passes skip_duplicates: true (ON CONFLICT DO NOTHING), so a conflicting row is skipped and the rest of the batch still inserts. A conflict here means the record is already registered, so skipping is correct for a consistency worker. On registry tables without a unique index on the model foreign key this is a no-op, so the change is safe for all registries and deploy orderings.
Resolves #630620 (closed)
References
- Issue: #630620 (closed)
- Spotted in review: !255707 (comment 3885273084)
- Related unique index issues: #606428 (closed), #606429 (closed), #606430 (closed)
Screenshots or screen recordings
Not applicable, backend-only change with no UI impact.
How to set up and validate locally
-
On a Geo secondary's Rails console, create a registry row for an existing model:
Geo::CiSecureFileRegistry.create!(ci_secure_file_id: <existing_id>) -
Insert a batch that includes the already-registered ID:
Geo::CiSecureFileRegistry.insert_for_model_ids([<existing_id>, <other_id>]) -
Before this change, this raised
ActiveRecord::RecordNotUniqueand inserted nothing. Now it skips the duplicate and inserts the remaining row.
Database review
The only SQL change: the bulk insert in insert_for_model_ids now ends with ON CONFLICT DO NOTHING RETURNING "id". No new queries, no schema changes, and the plan shape is the same as before (Values Scan -> Insert). The clause names no conflict target, so it applies to any unique index on the registry table - on registries without a unique index on the model FK it's a no-op.
Test setup - registry tables live on the Geo tracking DB, which isn't in the postgres.ai snapshot at all (Geo doesn't run on .com). So I created ci_secure_file_registry from the DDL in ee/db/geo/structure.sql (all 8 production indexes) and seeded 1M rows - 85% synced / 5% failed / 10% pending, with states and timestamps shuffled so nothing lines up with the PK to flatter the numbers. Ran ANALYZE after seeding. Geo::Secondary::RegistryConsistencyWorker inserts batches of up to 10,000 IDs, so that's the worst case tested.
The query (truncated to 2 tuples here - real batches carry up to 10,000):
INSERT INTO "ci_secure_file_registry"
("ci_secure_file_id","created_at","last_synced_at","retry_at","verified_at",
"verification_started_at","verification_retry_at","state","verification_state",
"retry_count","verification_retry_count","checksum_mismatch","verification_checksum",
"verification_checksum_mismatched","verification_failure","last_sync_failure")
VALUES
(11111, '2026-09-30 16:18:05', NULL, NULL, NULL, NULL, NULL, 0, 0, 0, 0, FALSE, NULL, NULL, NULL, NULL),
(22222, '2026-09-30 16:18:05', NULL, NULL, NULL, NULL, NULL, 0, 0, 0, 0, FALSE, NULL, NULL, NULL, NULL)
ON CONFLICT DO NOTHING
RETURNING "id"Before/after - the clause doesn't add cost. A plain INSERT already checks the unique index on every row (that's how it raises RecordNotUnique) - ON CONFLICT only changes what happens on a hit:
| Batch | Without ON CONFLICT |
With ON CONFLICT |
|---|---|---|
| 1,000 new rows | 23.5 ms | 25.1 ms |
| 10,000 new rows¹ | 166.7 ms | 122.0 ms |
The differences are within clone noise - the 10k batch was actually faster with the clause.
Duplicate handling (the new behaviour):
- 1 dupe + 999 new - 999 inserted, 26.9 ms (the exact race this MR fixes)
- 1,000 all dupes - 0 inserted, no error, 16.8 ms
- 10,000 all dupes - 0 inserted, no error, 78.0 ms
¹ Joe Bot rejects a ~1MB VALUES list, so the 10k plans use INSERT ... SELECT generate_series instead - same Insert node, only the source node differs.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.