Throttle packages last_downloaded_at updates
What does this MR do and why?
Every package download issued a synchronous UPDATE of
packages_packages.last_downloaded_at. For popular packages this produces a very
high volume of near-identical writes, visible in the SQL logs as floods of:
UPDATE "packages_packages" SET "last_downloaded_at" = $1 WHERE "packages_packages"."id" = $2Unlike other "last accessed" timestamps in the codebase, this write was not throttled.
This MR throttles the update to at most once per minute per record using a
conditional UPDATE — no Redis, no extra reads:
id_in(id)
.where(column.eq(nil).or(column.lt(THROTTLE_PERIOD.ago)))
.update_all(last_downloaded_at: Time.zone.now)Repeat downloads within the window match no row, so Postgres performs no heap or WAL
write and creates no dead tuples (the expensive part) — the statement becomes a cheap,
indexed no-op. The condition also covers the first-ever download (last_downloaded_at IS NULL), which a plain < ? comparison would miss.
The instance path (#touch_last_downloaded_at, used by npm/Helm/Maven/generic – the
record is already loaded) additionally short-circuits on the in-memory value, so hot
repeat downloads issue no statement at all, which is what removes them from the SQL
logs. The class-method path (Composer, id-only) relies on the conditional UPDATE.
THROTTLE_PERIOD = 1.minute. last_downloaded_at only needs coarse-grained accuracy (it
feeds cleanup policies measured in days/weeks), so this is harmless. The EE Geo-secondary
skip is preserved.
This follows the spirit of the existing last_used_at throttling in
Keys::LastUsedService / PersonalAccessTokens::LastUsedService, but keeps the throttle
in a single atomic, race-free SQL statement rather than adding a Redis dependency.
How to set up and validate locally
Download the same package file repeatedly within a minute and confirm only a single
last_downloaded_at write is performed:
touch /tmp/test.txt
curl -v -X PUT --header "PRIVATE-TOKEN: <PAT>" --upload-file /tmp/test.txt "https://gdk.test:3443/api/v4/projects/2/packages/generic/mypackage/1.0.1/file.txt"
curl -v --header "PRIVATE-TOKEN: <PAT>" "https://gdk.test:3443/api/v4/projects/2/packages/generic/mypackage/1.0.0/file.txt"
sleep 10
curl -v --header "PRIVATE-TOKEN: <PAT>" "https://gdk.test:3443/api/v4/projects/2/packages/generic/mypackage/1.0.0/file.txt"Then check bin/rails console. Packages::Package.last.last_downloaded_at should remain the same for a minute, no matter how many times you download.
Specs:
bundle exec rspec spec/models/concerns/packages/downloadable_spec.rb \
ee/spec/models/concerns/ee/packages/downloadable_spec.rb