Duplicate Geo JobArtifactRegistry Entries Leading to Sync Failures

Summary

Geo creates duplicate JobArtifactRegistry entries for the same artifact_id, which leads to synchronization failures. One registry entry synchronizes successfully, while the other consistently fails with a timeout error. Deleting both registry entries allows Geo to correctly create and synchronize a single JobArtifactRegistry entry.

Steps to reproduce

I was not able to reproduce this. We encountered this with a customer (ZD Ticket).

What is the current bug behavior?

  • Geo creates multiple JobArtifactRegistry entries for the same artifact_id.
  • One registry entry fails to synchronize, showing a timeout error, while the other completes successfully.
  • The error message is misleading, suggesting a timeout rather than indicating a duplicate entry.

What is the expected correct behavior?

  • Geo should create a single JobArtifactRegistry entry for each artifact_id.
  • If there is an issue, the error message should be more intuitive, such as indicating a duplicate entry rather than a timeout.

Relevant logs and/or screenshots

Geo::JobArtifactRegistry.where(artifact_id: 25055)
=> [
  #<Geo::JobArtifactRegistry:0x00007faa974f21c8
    id: 14106,
    created_at: Mon, 27 Nov 2023 17:43:07 UTC,
    artifact_id: 25055,
    state: 2,
    verification_state: 0,
    last_synced_at: Mon, 05 Aug 2024 11:15:04 UTC,
    verification_failure: nil,
    last_sync_failure: nil>,
  #<Geo::JobArtifactRegistry:0x00007faa974f2060
    id: 14105,
    created_at: Mon, 27 Nov 2023 17:43:07 UTC,
    artifact_id: 25055,
    state: 3,
    verification_state: 1,
    last_synced_at: Sat, 27 Jul 2024 18:47:08 UTC,
    verification_failure: "Verification timed out after 28800",
    last_sync_failure: "Verification failed with: Verification timed out after 28800">
]

Results of GitLab environment info

GitLab v16.11.6

Expand for output related to GitLab environment info 
GitLab v16.11.6

Workaround

  1. Open Rails console in secondary site

  2. Get number of model record IDs that have duplicates

    artifact_ids = Geo::JobArtifactRegistry.group(:artifact_id).having('COUNT(*) > 1').pluck(:artifact_id); artifact_ids.size
upload_ids = Geo::UploadRegistry.group(:file_id).having('COUNT(*) > 1').pluck(:file_id); upload_ids.size
package_file_ids = Geo::PackageFileRegistry.group(:package_file_id).having('COUNT(*) > 1').pluck(:package_file_id); package_file_ids.size

  3. Output the IDs in case we lose connection later

    puts 'BEGIN Artifact IDs', artifact_ids, 'END Artifact IDs'
puts 'BEGIN Upload IDs', upload_ids, 'END Upload IDs'
puts 'BEGIN Package File IDs', package_file_ids, 'END Package File IDs'

  4. Save the terminal output

  5. Delete all duplicates

    Geo::JobArtifactRegistry.where(artifact_id: artifact_ids).delete_all
Geo::UploadRegistry.where(file_id: upload_ids).delete_all
Geo::PackageFileRegistry.where(package_file_id: package_file_ids).delete_all

  6. Wait for background jobs to create the registry rows again and resync etc

Edited by Michael Kozono