npm metadata cache may miss changes made during a running rebuild (until_executing dedup race)

Summary

Packages::Npm::CreateMetadataCacheWorker has the same deduplication race that #631765 (closed) found in the Helm worker. The Helm worker was fixed in !259122 (merged). This issue covers fixing the npm worker the same way, as suggested in this review comment.

Current behavior

app/workers/packages/npm/create_metadata_cache_worker.rb:

deduplicate :until_executing
idempotent!

def perform(project_id, package_name)
  # ...
  ::Packages::Npm::CreateMetadataCacheService.new(project, package_name).execute
rescue StandardError => e
  Gitlab::ErrorTracking.log_exception(e, project_id: project_id, package_name: package_name)
end

app/services/packages/npm/create_metadata_cache_service.rb wraps the rebuild in ExclusiveLeaseGuard#try_obtain_lease. It uses the lease key packages:npm:create_metadata_cache_service:metadata_caches:<project_id>_<package_name> with DEFAULT_LEASE_TIMEOUT = 1.hour, the same pattern as Helm.

The race

  1. Job A starts rebuilding the metadata cache for (project_id, package_name) and takes the lease.
  2. With until_executing, the deduplication key is released as soon as job A starts. So a new event (a publish, deprecation, or package deletion) enqueues job B, and it is not deduplicated.
  3. Job B runs while job A is still running. It fails to get the lease, try_obtain_lease returns nil, and the job exits quietly without retrying.
  4. If job A read the package list before job B's change was committed, the cached metadata never includes that change, until some later event triggers another rebuild.

Result: npm install / npm view served from the metadata cache can show out-of-date versions, dist-tags, or deprecation state. Recently published versions can be missing.

Enqueue sites (all affected)

  • app/models/packages/npm/package.rb (perform_async(project_id, name))
  • app/services/packages/npm/process_package_file_service.rb (publish)
  • app/services/packages/npm/deprecate_package_service.rb (deprecate)
  • app/services/packages/mark_packages_for_destruction_service.rb (bulk_perform_async_with_contexts)
  • lib/api/helpers/packages/npm.rb

Bursts of publishes or deprecations for the same package, for example from CI pipelines publishing several versions, are the most likely to trigger it.

Proposal

Do what !259122 (merged) did for Helm:

deduplicate :until_executed, if_deduplicated: :reschedule_once

With this setting, events that arrive while a rebuild is running schedule exactly one more rebuild after the current one finishes. CreateMetadataCacheService is idempotent (find_or_build + update!), so running it again is safe.

Also consider:

  • Dedup TTL: with until_executed, the deduplication key has to last through queue wait plus the whole rebuild. Check the p99 run time for this worker and pass a longer ttl: if it is near the default. This was also raised for Helm in this comment.
  • Silent skip: CreateMetadataCacheService#execute returns nil when the lease is taken, and the worker ignores the result, so skipped rebuilds are never reported. See the related Helm issue #631919. It may be worth fixing both at once.
  • Specs: add assertions for get_deduplicate_strategy / get_deduplication_options in spec/workers/packages/npm/create_metadata_cache_worker_spec.rb, matching the Helm worker spec.

perform keeps the same signature, so this is safe to deploy under the Sidekiq compatibility guidelines.

Edited by 🤖 GitLab Bot 🤖