Skip to content

Add WAL transaction to the housekeeping manager

This MR integrates WAL transactioning system with the housekeeping manager. The manager now attempts to extract a transaction from the provided context. If a transaction is found, the housekeeping manager calls diverged housekeeping implementation in the transaction manager.

For every RPC request reaching Gitaly, a middleware initiates a transaction and injects it into the context of the request. As a result, the housekeeping manager can pick the transaction up from housekeeping invocations in RPC handlers.

Apart from that, Gitaly starts a cron-like housekeeping scheduler. This scheduler walks through the list of repositories and triggers housekeeping randomly. Thus, this MR also adds transaction initialization to the aforementioned scheduler.

The housekeeping manager skips some housekeeping tasks. Those tasks are deemed unnecessary after the transaction is enabled. Two significant tasks are:

  • Pruning objects. The traditional housekeeping manager prunes all objects exceeding a grace period. This period is to ensure an object is not removed while it's used by another request. WAL handles parallel requests in a very different way and unreachable objects are not included. Eventually, they'll be removed by a full repack. Thus, no need to call pruning independently.
  • Clean stale data. A WAL transaction copies meaningful data over when it is applied to the destination repository. All temporary files, dirs, locks, etc. are removed. Thus, there should be no stale data.

Manual tests

I performed multiple manual testing scenarios on GDK with Transaction enabled.

Scenario Before After Logs / Metrics
Run housekeeping tasks manually on the UI
Screenshot_2024-03-06_at_16.15.21
Screenshot_2024-03-06_at_16.20.04 Screenshot_2024-03-06_at_16.20.58 Repacked with FullWithCruft strategy Screenshot_2024-03-25_at_13.37.58
Push the repository N times - A secondary OptimizeRepository RPC is triggered Screenshot_2024-03-06_at_16.23.55 Screenshot_2024-03-06_at_16.23.50 Repacked with Geometric strategy Screenshot_2024-03-25_at_13.34.26 )
Daily optimizer Screenshot_2024-03-06_at_16.12.20 Screenshot_2024-03-06_at_16.12.29 Repacked with Geometric strategy Screenshot_2024-03-25_at_15.19.42
Run optimization on an object pool member Member
Screenshot_2024-03-06_at_15.43.05
Pool
Screenshot_2024-03-06_at_16.46.25
Member
Screenshot_2024-03-06_at_16.55.13
Pool (unchanged)
Screenshot_2024-03-06_at_16.55.18
Member (Repacked with FullWithCruft strategy - common objects are deduplicated, private objects are kept)
Screenshot_2024-03-25_at_15.25.54
Screenshot_2024-03-06_at_16.58.44
Pool unchanged
Pack references Screenshot_2024-03-06_at_17.05.50 Screenshot_2024-03-06_at_17.07.12 Screenshot_2024-03-06_at_17.07.14

What's not in this MR?

Edited by Quang-Minh Nguyen

Merge request reports