Loading
feat(indexer): lease archive downloads per Gitaly storage across pods
What does this MR do and why?
During INC-15163 one Gitaly node was saturated by archive fetches, because the only limit was per indexer pod and KEDA adds pods as the backlog grows. This puts a shared budget on concurrent archive downloads per Gitaly storage across all pods, and cancels a download if its slot is lost. The Rails change that returns the storage is a separate follow-up. Until then, a separate tenant-wide budget applies.
Related Issues
Closes #1391 (closed)
Testing
- Unit tests for the lease and for the resolver wiring.
- Integration tests against a real NATS with several independent pods: the cap holds across pods, a lost lease cancels the download, and a crashed pod's slot is recovered after the TTL.
mise lint:code,lint:fmtand the commit hooks pass.lint:docsreports one broken link in.gitlab/.argo/mr_template_doc_ai.md, which this MR does not touch.
Performance Analysis
Each download adds a few NATS KV operations plus a heartbeat every 10 s. Waiting downloads poll about every 500 ms, for at most 30 s, then retry on the redelivery cadence. Nothing changes on the parse or write path. The default budget of 4 per storage is a starting point and is tunable per tenant.
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered
storage_lease.rs:ArchiveLeases::runtakes one of the slot keys (archive.<base64url storage>.<n>, orarchive.tenant.<n>) in theindexing_locksbucket viaLockGuard, so it reuses the create-as-CAS, renew and expiry-takeover logic fromlocking.rs. The heartbeat (ttl/3) runs concurrently with the work. OnLockError::Lost, or when renewals fail untiltime_left()is zero, the work future is dropped.LockGuard::abandonis new: after a loss, or when the stored expiry has already passed at the end, the key may belong to another holder, so it is not deleted. Waiters start at a random slot and poll with jitter, and stop when the project-lock cancel token fires.- Fencing on the default path:
LocalRepositoryCache::extract_archivereads on a detachedspawn_blocking, so dropping the future did not stop the stream. It now sets a cancel flag on drop that the stream adapter checks, so the next read fails. A unit test streams a throttled archive and fails without the flag; an integration test does a real takeover (pod B wins the key with a new revision) against NATS. - Resolver:
repository_storagecomes fromproject_infothrough the existing 1 hCachingRepositoryService, which also covers push tasks. A lease wait that runs out returnsResolveError::NoArchiveSlot, mapped toFetched::NoArchiveSlotandIndexError::NoLane, so the handler returnsBackpressure, the end-of-run refund gives the attempt back while the project lock is still held, and norepository_fetchstage error is recorded. A lost project lock also stops the wait and logs that cause. That case still goes back to the queue, but the attempt stays counted, the same as a lost lock during the lane wait. Lease loss or a backend error retries throughARCHIVE_DOWNLOAD_RETRY. - Overload alignment (rebased after the 429/502/503/504 fix): the two capacity outcomes stay separate variants. A lease timeout is our own queueing, so it maps to
NoLane: the handler returnsBackpressure, which is always refunded while the project lock is held, and records no stage error. Archive overload is the upstream failing, so it keepsOverloaded: refund on redeliveries only,Processingnack, andstage=repository_fetch_overload. They never share a path, so nothing is refunded or counted twice. Overload is aGiveUpinside the retry closure, so the lease is dropped before the nack and no slot is held while waiting for redelivery. A resolver test covers the release. - Config:
archive_storage_budget(4),archive_tenant_budget(8, for unknown storage; independent key space, so a storage can see both budgets during a mixed rollout),archive_lease_ttl_secs(30, clamped to at least 1),archive_lease_max_wait_secs(30). 0 disables a budget. Schema, metrics catalog, observability, code indexing and ADR 018 docs updated.ProjectInfo.repository_storageisOption<String>withserde(default). - Metrics:
archive.lease.wait,archive.lease.slots_in_use(per pod, sum across pods),archive.duration,archive.lease.lost, labelled bystorage. - Trade-off: a lease waiter still holds a per-pod fetch slot, but only for the max wait (30 s). The NAK is not free: it spends a JetStream delivery (
max_deliver: 5,retry_interval60 s), so a storage saturated for several minutes drops the task from the stream and the project waits for the dispatcher cadence (up to the 1 h backfill window). Under sustained saturation the lease is dispatcher-paced retry, not a queue. Each cycle also costs two checkpoint writes, a project-lock acquire and up to about 60 polls of every slot key. The fetch-slot layer itself is not unit-tested; a handler test covers the bounded wait and the refund. - Follow-ups: waiters are not FIFO and still poll rather than watch the prefix. The project lock still deletes the new owner's key on drop, which
abandonavoids here. The archive lease now does the same when its run is dropped (for example by the job timeout) after its expiry. A revision-checked delete would also close the short window around a renew. The job-timeout drop benefits from the same extractor fix. Milestone to be set when out of draft. - Alternative considered: JetStream per-storage consumers with
max_ack_pending. It needs the storage in the subject at publish time, which no publisher has today.
Edited by Dmitry Gruzd