housekeeping: Claim stats cache key before walking
What does this MR do?
The housekeeping middleware walks a repository once per process to decide whether its file and
directory counts warrant forcing housekeeping. The walk was guarded by a sync.Map, but the key
was stored only after the walk returned, so every accessor RPC arriving while a walk was in flight
started its own duplicate walk of the same repository. On a repository with hundreds of thousands
of loose entries a walk takes tens of seconds, so the number of concurrent walks grew with the
request rate with nothing to bound it. In INC-14259 that put 33 GB of filepath.Walk allocations
on a single node, all for one repository.
The key is now claimed with LoadOrStore before the walk. Requests which lose the race skip the
statistics and proceed to the handler rather than blocking on the winner, so the walk stays off
their latency path.
Two details worth knowing when reading the diff:
- The claim is taken after the repository path is resolved, so repositories which are created
lazily — wikis, for instance — don't store and delete a key on every accessor RPC they receive
before they exist on disk. Resolving the path stats the storage, the repository and several of
its entries, so a plain
Loadshort-circuits already-claimed repositories before any of that work happens. - Claiming before the walk would otherwise mean a failed walk is never retried, since the key outlives the attempt. The claim is released from a deferred function whenever the walk doesn't complete, so a later request picks it up again.
Relevant issues
Closes #7429 (closed)