Geo: Skip duplicate rows when bulk inserting registries

What does this MR do and why?

Geo::BaseRegistry.insert_for_model_ids used bulk_insert! with the default on_duplicate: :raise. On registries with a unique index on the model foreign key, a race between Geo::RegistryConsistencyWorker and the sync path's find_or_initialize_by raised ActiveRecord::RecordNotUnique and aborted the whole batch (up to 10,000 rows). Because the batcher advances its cursor before the insert runs, the failed range was skipped and those records stayed unregistered and unsynced until the batcher wrapped around the whole table.

This MR passes skip_duplicates: true (ON CONFLICT DO NOTHING), so a conflicting row is skipped and the rest of the batch still inserts. A conflict here means the record is already registered, so skipping is correct for a consistency worker. On registry tables without a unique index on the model foreign key this is a no-op, so the change is safe for all registries and deploy orderings.

Resolves #630620 (closed)

References

Screenshots or screen recordings

Not applicable, backend-only change with no UI impact.

How to set up and validate locally

  1. On a Geo secondary's Rails console, create a registry row for an existing model:

    Geo::CiSecureFileRegistry.create!(ci_secure_file_id: <existing_id>)
  2. Insert a batch that includes the already-registered ID:

    Geo::CiSecureFileRegistry.insert_for_model_ids([<existing_id>, <other_id>])
  3. Before this change, this raised ActiveRecord::RecordNotUnique and inserted nothing. Now it skips the duplicate and inserts the remaining row.

Database review

The only SQL change: the bulk insert in insert_for_model_ids now ends with ON CONFLICT DO NOTHING RETURNING "id". No new queries, no schema changes, and the plan shape is the same as before (Values Scan -> Insert). The clause names no conflict target, so it applies to any unique index on the registry table - on registries without a unique index on the model FK it's a no-op.

Test setup - registry tables live on the Geo tracking DB, which isn't in the postgres.ai snapshot at all (Geo doesn't run on .com). So I created ci_secure_file_registry from the DDL in ee/db/geo/structure.sql (all 8 production indexes) and seeded 1M rows - 85% synced / 5% failed / 10% pending, with states and timestamps shuffled so nothing lines up with the PK to flatter the numbers. Ran ANALYZE after seeding. Geo::Secondary::RegistryConsistencyWorker inserts batches of up to 10,000 IDs, so that's the worst case tested.

The query (truncated to 2 tuples here - real batches carry up to 10,000):

INSERT INTO "ci_secure_file_registry"
  ("ci_secure_file_id","created_at","last_synced_at","retry_at","verified_at",
   "verification_started_at","verification_retry_at","state","verification_state",
   "retry_count","verification_retry_count","checksum_mismatch","verification_checksum",
   "verification_checksum_mismatched","verification_failure","last_sync_failure")
VALUES
  (11111, '2026-09-30 16:18:05', NULL, NULL, NULL, NULL, NULL, 0, 0, 0, 0, FALSE, NULL, NULL, NULL, NULL),
  (22222, '2026-09-30 16:18:05', NULL, NULL, NULL, NULL, NULL, 0, 0, 0, 0, FALSE, NULL, NULL, NULL, NULL)
ON CONFLICT DO NOTHING
RETURNING "id"

Before/after - the clause doesn't add cost. A plain INSERT already checks the unique index on every row (that's how it raises RecordNotUnique) - ON CONFLICT only changes what happens on a hit:

Batch Without ON CONFLICT With ON CONFLICT
1,000 new rows 23.5 ms 25.1 ms
10,000 new rows¹ 166.7 ms 122.0 ms

The differences are within clone noise - the 10k batch was actually faster with the clause.

Duplicate handling (the new behaviour):

¹ Joe Bot rejects a ~1MB VALUES list, so the 10k plans use INSERT ... SELECT generate_series instead - same Insert node, only the source node differs.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Scott Murray

Merge request reports

Loading
Loading