Schedule trailing project statistics refresh after lease expiry

What does this MR do and why?

Project statistics events that arrive while UpdateProjectStatisticsWorker holds the ProjectCacheWorker lease are no longer dropped. Behind the project_statistics_trailing_refresh flag (gitlab_com_derisk, off by default, project actor), ProjectCacheWorker#update_statistics now makes sure one UpdateProjectStatisticsWorker runs when the current lease expires. The 15-minute rate limit per project and statistics set stays in place.

Why this happens, and why this fix
  • ProjectCacheWorker#update_statistics debounces refreshes with a 15-minute lease and schedules a delayed UpdateProjectStatisticsWorker.
  • That delayed worker takes the same lease again for 15 minutes and schedules nothing. Any event in that second window failed try_obtain in ProjectCacheWorker and was dropped. On a quiet project the statistic stayed stale for good. The issue reports this for container_registry_size on about 20% of a customer's projects.
  • The delayed worker re-takes the lease on purpose. c51569d2 added it to cap expensive repository_size refreshes at one per 15 minutes. So this MR keeps that lease instead of removing it, and adds a trailing refresh:
    • Whether or not it gets the lease, update_statistics schedules UpdateProjectStatisticsWorker for just after the lease expires (remaining TTL plus a 5 second margin for clock skew).
    • A :trailing lease keeps this to one scheduled job per lease period. Redis rounds TTL to the nearest second, and the key is set in a separate call, so it uses ttl - 2 to expire well before the main lease. The next period can then schedule its own job.
  • Every refresh still needs a successful try_obtain on the same lease, so refreshes cannot exceed one per 15 minutes per project and statistics set. Extra jobs can only be no-ops.
  • With the flag off, the code keeps the exact old behavior.
Why a feature flag

The cap does not change, but events that used to be dropped now cause a real refresh. For repository_size that is a Gitaly call, so total refresh load on GitLab.com goes up somewhat on busy projects. The flag allows a gradual rollout and a quick rollback. Rollout issue: #633864

Alternatives considered
  • Do not re-take the lease in UpdateProjectStatisticsWorker (option 1 in the issue): smallest diff, but it removes the 2022 rate limit.
  • idempotent! plus deduplicate :until_executing, including_scheduled: true on UpdateProjectStatisticsWorker: avoids the second lease, but deduplication is set per worker class, so the project flag can't control it, and it would change the flag-off path too. On master, an event that takes the lease before the delayed job starts schedules a job for 15 minutes later. That job recovers events from the next lease period. With deduplication, that job is dropped as a duplicate of the pending one.
Known gaps, not addressed here
  • If Projects::UpdateStatisticsService raises inside UpdateProjectStatisticsWorker, its retries hit the lease it already holds and do nothing. This is pre-existing. With the flag on, the impact is smaller, because events in that period schedule the next trailing refresh.
  • A push refreshes repository_size under two lease keys (repository_size and commit_count:repository_size), so a busy project can get two refreshes per 15 minutes. This is pre-existing and unchanged.

References

Screenshots or screen recordings

Not a UI change. Output of the console steps below on a GDK:

Before (flag off, same as master) After (flag on)
T0+20m: event C, scheduled UpdateProjectStatisticsWorker jobs=0 T0+20m: event C, scheduled UpdateProjectStatisticsWorker jobs=1
T0+30m+: stored=1230109 live=2278685 => STALE (bug) T0+30m+: stored=2278685 live=2278685 => IN SYNC

How to set up and validate locally

The steps follow the timeline from the issue. To avoid waiting 30 minutes, they stop Sidekiq so scheduled jobs stay inspectable, and they simulate lease expiry by deleting the lease keys. They use lfs_objects_size because GDK can change it without a container registry. The code path is the same for container_registry_size.

  1. Stop Sidekiq: gdk stop rails-background-jobs

  2. Open a Rails console: gdk rails console

  3. Paste the setup:

    project = Project.find_by_full_path('gitlab-org/gitlab-test')
    stats = %w[lfs_objects_size]
    key = "project_cache_worker:#{project.id}:lfs_objects_size"
    expire_lease = -> { [key, "#{key}:trailing"].each { |k| Gitlab::ExclusiveLease.cancel(k, Gitlab::ExclusiveLease.get_uuid(k)) } }
    scheduled = -> { Sidekiq::ScheduledSet.new.select { |j| j.klass == 'UpdateProjectStatisticsWorker' && j.args[1] == project.id } }
    run_scheduled = -> { scheduled.call.each { |j| UpdateProjectStatisticsWorker.new.perform(*j.args); j.delete } }
    live = -> { LfsObject.joins(:lfs_objects_projects).where(lfs_objects_projects: { project_id: project.id }).sum(:size) }
    stored = -> { project.statistics.reload.lfs_objects_size }
    
    expire_lease.call
    scheduled.call.each(&:delete)
    project.statistics.refresh!(only: [:lfs_objects_size])
  4. Run the timeline. First run it as is to see the bug, then run Feature.enable(:project_statistics_trailing_refresh, project) and run the setup and timeline again:

    # T0: event A. Lease taken, immediate refresh, delayed worker scheduled
    ProjectCacheWorker.new.perform(project.id, [], stats)
    
    # ~T0+15m: lease expires, the delayed worker runs and takes the lease again
    expire_lease.call
    run_scheduled.call
    Gitlab::ExclusiveLease.new(key, timeout: 1).ttl # => ~900
    
    # ~T0+20m: event C. A new LFS object is linked to the project
    lfs = LfsObject.create!(oid: SecureRandom.hex(32), size: 1.megabyte)
    link = LfsObjectsProject.create!(project: project, lfs_object: lfs)
    ProjectCacheWorker.new.perform(project.id, [], stats)
    scheduled.call.size # flag off: 0, flag on: 1
    
    # ~T0+30m: lease expires and every scheduled job runs, then the project goes quiet
    expire_lease.call
    run_scheduled.call
    [stored.call, live.call] # flag off: stored is 1 MiB behind live. Flag on: equal
  5. Clean up:

    link.destroy!
    lfs.destroy!
    expire_lease.call
    scheduled.call.each(&:delete)
    project.statistics.refresh!(only: [:lfs_objects_size])
    Feature.remove(:project_statistics_trailing_refresh)
  6. Start Sidekiq again: gdk start rails-background-jobs

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Reviews run before opening
  • Correctness, integration with callers, a security review of the new code, and an adversarial review of the surrounding code. No security findings in the change or in the surrounding workers, their callers, or the registry notification endpoint.
  • Changes made from review findings: the TTL rounding fix (ttl - 2), a 5 second scheduling margin, a spec for the "lease gone" path, and the feature flag.
  • No changelog entry, because the change is behind a default-off flag.

🤖 Generated with Claude Code

Edited by Niklas Janz

Merge request reports

Loading
Loading