[Feature flag] Enable writing cruft packs
What
Enable the :gitaly_write_cruft_packs
feature flag, which enables writing cruft packs in our repository housekeeping logic. This rolls out #4351 (closed).
In order to avoid races during garbage collection, Git will only delete unreachable objects after a specific grace period. This is why repacking a repository will explode all unreachable objects into loose objects, so that Git can track the last time each of these unreachable objects has been accessed. This is an important problem though, as loose objects cannot be stored deltified and will potentially slow down Git commands when there are too many of them. But storing them in a simple packfile does not fly because the access time of all objects contained in that packfile would be freshened whenever a single of them is being accessed.
To solve this issue, Git has introduced cruft packs in Git v2.37.0. This
mechanism allows Git to continue storing unreachable objects in a pack
that is annotated with a .mtimes
data structure. This data structure
tracks per-object access times that can be updated separately whenever
any of the objects is being accessed.
There are some more advantages that cruft packs give us that are more Gitaly-specific:
- We can use it as a stepping stone to efficiently compute a
repository's size while excluding unreachable objects that are
part of a cruft pack.
- It gives us better insight into truly unreachable objects and
reachable ones. This has been an issue with our current metrics
which only discern recent and unreachable objects. Those have
caused us to underestimate the amount of unreachable objects.
- In the general case, we will likely be able to punt on running
git-prune(1) more often as repositories typically would not have
loose objects anymore.
Cruft packs are generated at the time git-repack(1) is executed, and furthermore only when doing a full repack. What it does in that case is that it will not only write a single packfile, but two packfiles that are segregated into reachable and unreachable objects. Furthermore, the command will also remove any objects that had been part of a cruft pack and which have last been accessed before the grace period.
Owners
- Team: Gitaly
- Most appropriate slack channel to reach out to:
#g_gitaly
- Best individual to reach out to: NAME
Expectations
What release does this feature occur in first?
What are we expecting to happen?
We should see that Gitaly starts to write cruft packs. No user-visible change in behaviour should occur. We might see a small decrease in storage space in repositories that have a lot of unreachable objects.
What might happen if this goes wrong?
Repository corruption is a possibility if there was a bug in the way cruft packs are generated.
What can we monitor to detect problems with this?
https://dashboards.gitlab.net/d/Z2xwZIP7k/gitaly-housekeeping-statistics?orgId=1
Roll Out Steps
-
Enable on staging -
Is the required code deployed on staging? (howto) -
Enable on staging (howto) -
Add featureflagstaging to this issue (howto) -
Test on staging (howto) -
Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
-
-
Enable on production -
Is the required code deployed on production? (howto) -
Progressively enable in production (howto) -
Add featureflagproduction to this issue -
Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
-
-
Default-enable the feature flag (optional, only required if backwards-compatibility concerns exist) -
Wait for release containg default-disabled feature flag. -
Change the feature flag to default-enabled (howto) -
Wait for release containing default-enabled feature flag.
-
-
Remove feature flag
Please refer to the documentation of feature flags for further information.