Skip to content

Conan download failed under high load (download/upload concurrent operations).

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Conan Package Registry Race Conditions Under Concurrent Load

GitLab Version: v18.2.1-ee
Environment: Deployed in Kubernetes, for clear experiment autoscaling was disabled.

Problem

During concurrent conan upload / conan download operations, packages fail with:

  • ERROR: catch2/3.9.0: ERROR: Exception: <class 'conan.errors.ConanException'> Corrupted catch2/3.9.0:276f2e669dbe0389bb1d64610db72d3522e37f2e in 'gitlab' remote: no conanmanifest.txt
  • 404 Not Found from /api/v4/projects/{id}/packages/conan/v2/conans/.../latest endpoint

Suspected Technical Causes

⚠️ Note: Based on static code analysis - runtime debugging needed to confirm.

1. /latest Endpoint Race Condition

Location: lib/api/conan/v2/project_packages.rb:318

revision = package_revisions.order_by_id_desc.first
not_found!('Revision') unless revision.present?

Queries for revisions before concurrent upload transactions complete, causing temporary 404s.

2. Async conaninfo.txt Processing

Location: app/services/packages/conan/create_package_file_service.rb:38-40

if package_file.file_name == ::Packages::Conan::FileMetadatum::CONANINFO_TXT
  ::Packages::Conan::ProcessPackageFileWorker.perform_async(package_file.id)
end

Files marked as available (status: :default) before background metadata extraction completes.

3. Concurrent Revision Creation Conflicts

Database upsert operations in upsert_recipe_revision_service.rb may conflict when multiple processes create identical revisions simultaneously.

Investigation Needed

  1. Enable detailed logging around package creation/revision operations
  2. Monitor Sidekiq queue lag during load testing
  3. Add timing instrumentation to /latest endpoint
  4. Check PostgreSQL transaction isolation behavior

Potential Quick Fixes

  • Add retry logic to /latest endpoint for transient 404s
  • Use status: :processing for conaninfo.txt until extraction completes
  • Consider database advisory locks for revision creation

Reproducing

Run multiple concurrent sessions of conan install . ; conan upload * against the same GitLab project.

Related to #519741 (closed)

@oceane_scania @dmeshcharakou Maybe I am wrong with the guess, but this is the last issue that stopping us to migrate fully on conan package registry. Might impact users on scale. Would appreciate your help.

Edited by 🤖 GitLab Bot 🤖