Geo - Replicate Projects::ImportExport::RelationExportUpload

What does this MR do and why?

This code change adds support for replicating and verifying Projects::ImportExport::RelationExportUpload uploads in GitLab's Geo feature (which keeps multiple GitLab instances synchronized).

The changes create a new database table to track the synchronization status of relation export upload files uploaded during project imports/exports. This includes adding verification states to ensure files are properly copied between different GitLab servers, along with checksums to verify file integrity.

The update also adds new monitoring metrics so administrators can track how many RelationExportUpload uploads have been successfully synchronized, failed, or are pending verification across their GitLab instances. Additionally, it updates the API documentation to include these new RelationExportUpload upload statistics in the Geo status reports.

This enhancement ensures that relation export uploads are properly backed up and synchronized across multiple GitLab installations for disaster recovery and performance purposes.

🚨 Heads-up for SREs / Database reviewers

This MR re-introduces the migrations that were reverted in !234263 (merged) to address gl-infra/production#21955 (INC-9725). The original MR !231557 (merged) failed on production-canary because db/migrate/20260414204422_create_project_import_export_relation_export_upload_upload_states.rb combined create_table with an add_concurrent_foreign_key … :projects call; the FK exhausted with_lock_retries against the hot projects table and blocked deploys for ~3 hours.

What changed in this re-introduction

The FK to projects has been moved out of the regular migration into a post-deployment migration: db/post_migrate/20260506140004_add_foreign_key_on_piere_upload_upload_states_to_projects.rb.

Plan if the FK migration fails on gprd

Per the discussion on the original MR (note, agreed with @ahegyi):

1. Mark 20260506140004 as successfully executed on gprd via the standard "manually mark migration as executed" flow.
2. Open a change request to add the FK manually during an off-peak window (no traffic on projects, plenty of lock-retry budget).

Migration order

Regular (pre-deploy):

• 20260506140000 — create project_import_export_relation_export_upload_upload_states table (no FKs).
• 20260506140001 — install sharding-key trigger on the new table.

Post-deploy:

• 20260506140002 — unique index on project_import_export_relation_export_upload_uploads(id) partition.
• 20260506140003 — FK from _upload_states to the _uploads partition.
• 20260506140004 — FK from _upload_states to projects (the one to watch).

Geo:

• 20260506135959 — create project_import_export_relation_export_upload_upload_registry table.

References

Relates to #589916 (closed)

How to set up and validate locally

Prerequisites

A working Geo GDK setup with both primary and secondary running. Follow the Geo development setup guide.

1. Run database migrations

rails db:migrate        # on the primary
rails db:migrate:geo    # on the secondary

2. Enable the feature flags on the primary

# In Rails console on the primary
Feature.disable(:geo_upload_replication)
Feature.enable(:geo_project_import_export_relation_export_upload_upload_replication)
Feature.enable(:geo_project_import_export_relation_export_upload_upload_force_primary_checksumming)

3. Create test data on the primary

Trigger a project export via the UI (Project > Settings > General > Advanced > Export project), or use the Rails console:

# In Rails console on the primary
user = User.admins.first
project = Project.first

project.add_export_job(current_user: user)

If Sidekiq isn't running in the env (rare in GDK), you can run it inline:

user = User.admins.first
project = Project.first

Sidekiq::Testing.inline! do
  project.add_export_job(current_user: user)
end

Wait a few seconds for Sidekiq to run CreateRelationExportsWorker → RelationExportWorker, then verify the upload exists in the project_import_export_relation_export_upload_uploads partition:

Geo::ProjectImportExportRelationExportUploadUpload.count # Should be > 0

Note: add_export_job calls check_duplicate_export!, so if a previous export is still started/queued you'll need to wait for it to finish (or pick a different Project) before re-running.

4. Verify checksumming on the primary

Wait for the verification worker to process, or trigger it manually:

# In Rails console on the primary
record = Geo::ProjectImportExportRelationExportUploadUpload.first
record.replicator.verify
record.project_import_export_relation_export_upload_upload_state.reload
record.project_import_export_relation_export_upload_upload_state.verification_state
# Should be 2 (verification_succeeded)

5. Verify replication on the secondary

Once the upload is created on the primary, Geo will automatically replicate it to the secondary. Check the sync status in the secondary Rails console:

# In Rails console on the secondary
Geo::ProjectImportExportRelationExportUploadUploadRegistry.count
# Should be > 0

registry = Geo::ProjectImportExportRelationExportUploadUploadRegistry.last
registry.state
# Should be 2 (synced)

If the registry is empty or not yet synced, you can manually trigger sync:

# In Rails console on the secondary
Geo::ProjectImportExportRelationExportUploadUploadReplicator.new(
  model_record_id: Geo::ProjectImportExportRelationExportUploadUpload.first.id
).sync

6. Verify verification on the secondary

# In Rails console on the secondary
registry = Geo::ProjectImportExportRelationExportUploadUploadRegistry.last
registry.reload
registry.verification_state
# Should be 2 (verification_succeeded)

7. Test GraphQL API on the secondary

Note: You must be logged in as an admin user. Non-admin users will get null for Geo-related queries.

Note: When querying from the secondary's GraphQL explorer, add a custom header REQUEST_PATH with the value /api/v4/geo/node_proxy/{node_id}/graphql

Open the GraphQL explorer on the secondary instance (http://<secondary-url>/-/graphql-explorer) and run:

query {
  geoNode {
    name
    primary
    projectImportExportRelationExportUploadUploadRegistries {
      nodes {
        id
        state
        verificationState
        projectImportExportRelationExportUploadUploadId
        lastSyncedAt
        verifiedAt
      }
    }
  }
}

Expected result: you should see registry entries with state: "SYNCED" and verificationState: "VERIFIED".

8. Verify Geo Sites API

Check the Geo Sites API includes the new RelationExportUpload upload statistics:

curl --header "PRIVATE-TOKEN: <your-token>" "http://<primary-url>/api/v4/geo_sites/status"

Look for the new fields in the response:

• project_import_export_relation_export_upload_upload_count
• project_import_export_relation_export_upload_upload_checksummed_count
• project_import_export_relation_export_upload_upload_checksum_failed_count
• project_import_export_relation_export_upload_upload_synced_count
• project_import_export_relation_export_upload_upload_failed_count
• project_import_export_relation_export_upload_upload_registry_count
• project_import_export_relation_export_upload_upload_synced_in_percentage
• project_import_export_relation_export_upload_upload_verified_in_percentage

9. Verify Geo admin page

Visit /admin/geo/sites on the secondary and confirm that "Project Import Export Relation Export Upload Upload" appears as a new data type with replication and verification progress.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.