Direct transfer fails with "null value in column project_relation_export_id violates not-null constraint"

Summary

Direct transfer (migration) fails with a database constraint violation error when the RelationExportWorker is retried after a failed upload attempt.

Error Message 

PG::NotNullViolation: ERROR: null value in column "project_relation_export_id" of relation "project_relation_export_uploads" violates not-null constraint

Steps to Reproduce

  1. Perform a direct transfer/migration of a project
  2. First export attempt creates an upload record successfully
  3. Upload fails due to network issue, timeout, or storage problem
  4. Sidekiq retries the worker (up to 6 times)
  5. Error occurs on retry when trying to create a new upload record

Expected Behavior

Direct transfer should handle retries gracefully, reusing or properly replacing the upload record without constraint violations.

Actual Behavior

The retry fails with a NOT NULL constraint violation because Rails tries to nullify the existing upload's project_relation_export_id when building a new upload.

Root Cause Analysis

The bug occurs on worker retry, not the first attempt:

  1. First attempt: relation_export.build_upload creates an upload record successfully
  2. Upload fails: Network issue, timeout, or other error during file upload
  3. Worker retries: Sidekiq retries the job (configured for up to 6 retries)
  4. Second attempt: relation_export.build_upload is called again
  5. Rails behavior: When building a new has_one association, Rails tries to nullify the existing upload's project_relation_export_id to replace it
  6. Constraint violation: The NOT NULL constraint prevents this nullification

Code Location

The bug is in app/services/projects/import_export/relation_export_service.rb:

def upload_compressed_file
  upload = relation_export.build_upload  # ← Fails on retry!
  File.open(archive_file_full_path) { |file| upload.export_file = file }
  upload.save!
end

Why Manual Export Works

Manual export steps execute synchronously without retries, so the issue never occurs.

Affected Code

  • Service: app/services/projects/import_export/relation_export_service.rb
  • Model: app/models/projects/import_export/relation_export_upload.rb
  • Worker: app/workers/projects/import_export/relation_export_worker.rb
  • Database: db/structure.sql (table: project_relation_export_uploads)

Solution (Implemented in !213686 (merged))

Reuse existing upload record on retry instead of trying to replace it:

def upload_compressed_file
  # Reuse existing upload if present (from previous failed attempt)
  # or build a new one. This avoids the NOT NULL constraint violation.
  upload = relation_export.upload || relation_export.build_upload
  File.open(archive_file_full_path) { |file| upload.export_file = file }
  upload.save!
end

Why This Approach is Better

Instead of destroying and recreating (DELETE + INSERT):

relation_export.upload&.destroy  # ❌ Inefficient
upload = relation_export.build_upload

We reuse the existing record (single UPDATE or INSERT):

upload = relation_export.upload || relation_export.build_upload  # ✅ Efficient

Benefits:

  1. Performance: Single UPDATE operation instead of DELETE + INSERT
  2. Data integrity: Preserves upload ID and audit trail (created_at timestamp)
  3. Simplicity: Cleaner, more idiomatic Rails code
  4. Database load: Fewer operations, less lock contention

Additional Changes

  • Add optional: false to belongs_to :relation_export for validation
  • Add specs to verify retry scenario works correctly

Workaround

Use manual export steps as documented in: https://docs.gitlab.com/user/project/settings/import_export_troubleshooting/#manually-execute-export-steps

Environment

  • Affected versions: GitLab 17.x - 18.x (any version with the current code structure)
  • Feature flag: ENABLE_STORE_EXPORT_FILE_AFTER_COMMIT=true (default)
  • Database: PostgreSQL (all supported versions)

Related References

  • Merge Request: !213686 (merged)
  • Migration adding NOT NULL constraint: db/post_migrate/20250606135453_add_project_relation_export_uploads_project_id_not_null.rb
  • CarrierWave callback configuration: Lines 23-27 in relation_export_upload.rb

Verification

You can test the fix in Rails console:

# Create test export
relation_export = Projects::ImportExport::RelationExport.create!(...)
service = Projects::ImportExport::RelationExportService.new(relation_export, user, jid)

# First attempt
service.execute
upload_id_1 = relation_export.upload.id

# Simulate retry
relation_export.update!(status: 0, jid: nil)
service.execute
upload_id_2 = relation_export.upload.id

# Verify same record was reused
upload_id_1 == upload_id_2  # => true ✅
Edited by David Wainaina