Feasibility Study

Background

In this MR, a brilliant idea was proposed to use OverlayFS for serving snapshots in Gitaly transactions.

Currently, Gitaly uses a "deep-clone" strategy to take a snapshot of a repository: it walks the tree, creates the corresponding directory structure, and hard-links all files to the new destination. Unfortunately, this approach does not scale well. By contrast, OverlayFS can create snapshots in constant time, regardless of repository size.

This issue is a feasibility study on whether OverlayFS can be used for snapshotting. There are several key questions we need to answer before declaring it viable. Until we have good solutions for those, jumping straight into benchmarking or building a PoC may be premature and potentially a waste of time.

Key Question 1: Lower directory must be read-only

related thread

Overlayfs docs explicitly warn that changing a lowerdir while it is mounted can lead to undefined behavior:

Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed. If the underlying filesystem is changed, the behavior of the overlay is undefined, though it will not result in a crash or deadlock.

As pointed out here:

Imagine two concurrent transactions:

• Transaction 1 and Transaction 2 both create snapshots on top of the same lower state.
• Transaction 1 commits first and we merge its new packfile back into the base.
• Transaction 2 is still live on the pre-commit snapshot, but a getdents64 (what ls ultimately calls) might or might not see the new packfile from Transaction 1, depending on kernel internals.

So the open question is: when taking a snapshot, what should be the actual “base” of the lower directory?

• Thread for possible solutions