npm metadata cache may miss changes made during a running rebuild (until_executing dedup race)
Summary
Packages::Npm::CreateMetadataCacheWorker has the same deduplication race that #631765 (closed) found in the Helm worker. The Helm worker was fixed in !259122 (merged). This issue covers fixing the npm worker the same way, as suggested in this review comment.
Current behavior
app/workers/packages/npm/create_metadata_cache_worker.rb:
deduplicate :until_executing
idempotent!
def perform(project_id, package_name)
# ...
::Packages::Npm::CreateMetadataCacheService.new(project, package_name).execute
rescue StandardError => e
Gitlab::ErrorTracking.log_exception(e, project_id: project_id, package_name: package_name)
endapp/services/packages/npm/create_metadata_cache_service.rb wraps the rebuild in ExclusiveLeaseGuard#try_obtain_lease. It uses the lease key packages:npm:create_metadata_cache_service:metadata_caches:<project_id>_<package_name> with DEFAULT_LEASE_TIMEOUT = 1.hour, the same pattern as Helm.
The race
- Job A starts rebuilding the metadata cache for
(project_id, package_name)and takes the lease. - With
until_executing, the deduplication key is released as soon as job A starts. So a new event (a publish, deprecation, or package deletion) enqueues job B, and it is not deduplicated. - Job B runs while job A is still running. It fails to get the lease,
try_obtain_leasereturnsnil, and the job exits quietly without retrying. - If job A read the package list before job B's change was committed, the cached metadata never includes that change, until some later event triggers another rebuild.
Result: npm install / npm view served from the metadata cache can show out-of-date versions, dist-tags, or deprecation state. Recently published versions can be missing.
Enqueue sites (all affected)
app/models/packages/npm/package.rb(perform_async(project_id, name))app/services/packages/npm/process_package_file_service.rb(publish)app/services/packages/npm/deprecate_package_service.rb(deprecate)app/services/packages/mark_packages_for_destruction_service.rb(bulk_perform_async_with_contexts)lib/api/helpers/packages/npm.rb
Bursts of publishes or deprecations for the same package, for example from CI pipelines publishing several versions, are the most likely to trigger it.
Proposal
Do what !259122 (merged) did for Helm:
deduplicate :until_executed, if_deduplicated: :reschedule_onceWith this setting, events that arrive while a rebuild is running schedule exactly one more rebuild after the current one finishes. CreateMetadataCacheService is idempotent (find_or_build + update!), so running it again is safe.
Also consider:
- Dedup TTL: with
until_executed, the deduplication key has to last through queue wait plus the whole rebuild. Check the p99 run time for this worker and pass a longerttl:if it is near the default. This was also raised for Helm in this comment. - Silent skip:
CreateMetadataCacheService#executereturnsnilwhen the lease is taken, and the worker ignores the result, so skipped rebuilds are never reported. See the related Helm issue #631919. It may be worth fixing both at once. - Specs: add assertions for
get_deduplicate_strategy/get_deduplication_optionsinspec/workers/packages/npm/create_metadata_cache_worker_spec.rb, matching the Helm worker spec.
perform keeps the same signature, so this is safe to deploy under the Sidekiq compatibility guidelines.
Related
- Helm bug: #631765 (closed)
- Helm fix: !259122 (merged)
- Helm silent-skip follow-up: #631919
reschedule_onceoption: #342123 (closed)- Docs: https://docs.gitlab.com/development/sidekiq/idempotent_jobs/#until-executed