Replace manifest pointer with generation numbers in Scaling Git design

backends for storing references and objects. The repository's persistent state

is a set of immutable, content-addressed artifacts. An immutable manifest tells

us which of these artifacts are currently supposed to be active. The currently

active manifest is referenced by the mutable manifest pointer. This design

active manifest is named by the repository's current generation. This design

allows for consistent reads and atomic updates, and multiple Git processes

running in the same repository can act on different versions thereof.

Gitaly is the orchestrator: it pulls the artifacts named by the active manifest

onto local disk, runs Git against that cache, publishes new artifacts back to

object storage, and advances the pointer with a compare-and-swap (CAS).

object storage, and activates the new manifest by conditionally creating the

next generation object.

As object storage holds the authoritative state, any node can serve any

repository after a cache fill, and nodes can be added on demand and torn down

  performance (cold cache), but it does not cause data loss or unavailability.

- A repository's read workload scales horizontally across nodes.

- Reads are observed as a consistent, point-in-time snapshot.

- Publishing a write is atomic and isolated, serialized by an atomic

  compare-and-swap on the manifest pointer.

- Publishing a write is atomic and isolated, serialized by a conditional,

  create-only write of the next generation object.

- The overhead of the local working set stays small: a node fetches only those

  artifacts required for a given repository, and prunes old data that is not

  referenced anymore.

  more[More Gitaly nodes ...]

  subgraph os[Object storage - source of truth]

    pointer[<b>Manifest pointer</b><br/>mutable<br/>currently active manifest]

    generations[<b>Generations</b><br/>mutable<br/>ordered list of manifests]

    artifacts[<b>Artifacts</b><br/>immutable<br/>manifests, packs, reftables, indices]

  end

  router -.-> more

  gitaly -->|spawn| git

  git -->|local file I/O only| cache

  gitaly -->|resolve / update pointer| pointer

  gitaly -->|resolve / write new generation| generations

  gitaly -->|prefetch| artifacts

  artifacts -->|fetch missing artifacts| cache

  gitaly -->|upload new artifacts| artifacts

- **Object storage** is the single source of truth for repository data. For each

  repository it holds a set of immutable, content-addressed artifacts (manifest

  bodies, packfiles, reftables, indices) and a single mutable manifest pointer

  that names the currently active manifest. The pointer is only ever advanced

  via an atomic compare-and-swap.

  bodies, packfiles, reftables, indices) and a `generations/` prefix that

  provides an ordered sequence of manifests that had been active that names the

  currently active manifest. The pointer is only ever advanced via an atomic

  compare-and-swap.

- **Git** provides the MVCC reference and object backends. It performs no

  network I/O of its own: it only reads and writes local files, and the set of

  files it sees is dictated by the active manifest. Pinning a Git process to a

Note that eventually, once the canonical source of truth sits in object storage,

Gitaly will never use `<commondir>/mvcc/manifest` anymore. Instead, it is

expected to always resolve the version with a GET request and then export it via

either `GIT_MVCC_MANIFEST` or by writing it into `GIT_MVCC_MANIFEST_PATH`.

expected to always resolve the version from the repository's current generation

in object storage and then export it via either `GIT_MVCC_MANIFEST` or by

writing it into `GIT_MVCC_MANIFEST_PATH`.

When neither `GIT_MVCC_MANIFEST` nor `GIT_MVCC_MANIFEST_PATH` are set, then Git

uses the pointer in `<commondir>/mvcc/manifest`.

### Object storage as source of truth

For each repository, object storage holds the full artifact set together with a

durable manifest-pointer key. Gitaly resolves this durable pointer at the start

of every RPC and never treats a local cache pointer as authoritative.

durable `generations/` prefix that provides an ordered sequence of manifests

that had been active, where the current manifest is one that sorts first in

lexicographic order. Gitaly resolves the current generation at the start of

every RPC and never treats a local cache pointer as authoritative.

Each generation is stored as its own key under the `generations/` prefix:

```text

<repository>/

  generations/<generation>          content: hash of the active manifest

  generations/<generation>.<hash>   optimization: manifest hash in the key

  manifests/<hash>                  immutable manifest bodies

  ...

```

The generation number is a zero-padded hexadecimal number that counts _down_

from `UINT64_MAX`. The current generation thus always has the lexicographically

smallest key, so a `List(prefix="generations/", limit=1)` call returns the

current generation as its first result. The content of the generation object is

the hash of the manifest that is active at that generation.

Activating a new manifest resolves the current generation and then performs a

conditional, create-only write of the next generation object. As the write fails

if the key already exists, concurrent writers racing for the same generation are

serialized: exactly one of them succeeds, and the others observe a conflict.

As an optimization, a writer additionally stores a

`generations/<generation>.<hash>` key, which embeds the manifest hash in its

name, after it has successfully written the generation object itself. With a

`List(limit=2)` call, a reader can thus resolve both the current generation and

its manifest hash in a single roundtrip. This is not race-free: a concurrent

reader may observe the generation object before the hash-suffixed key has been

written. In that case it falls back to reading the generation object itself to

resolve the manifest hash.

Should the generation number ever underflow, the scheme rolls over: keys of the

new epoch are prepended with a reserved prefix character that sorts

lexicographically before all hexadecimal digits, and every subsequent rollover

prepends one more such character. Keys of this new epoch thus keep sorting

first.

{{% alert %}}

An earlier iteration of this design used a single mutable manifest-pointer key

that was advanced with a compare-and-swap. This was abandoned because object

storage providers rate-limit updates to the same key (Google Cloud Storage for

example allows only one update per second per object).

{{% /alert %}}

There are a couple of requirements against the object storage provider:

- It must support single-key PUTs with read-after-write consistency.

- It must support conditional updates of the pointer key.

- It must support conditional, create-only PUTs of generation objects.

- It must support listing keys by prefix in lexicographic order with a limit.

- It must support conditional deletes to make housekeeping safe.

- It must support updating ETags of objects.

  participant Git as Git

  C->>G: read RPC

  G->>S: GET manifest pointer

  G->>S: List(prefix="generations/", limit=2)

  S-->>G: <generation> [+ <generation>.<version>]

  opt hash-suffixed key not yet visible

    G->>S: GET generations/<generation>

    S-->>G: <version>

  end

  opt manifest body not cached

    G->>S: GET manifests/<version>

    S-->>G: body

  G-->>C: response

```

Gitaly resolves the current generation with a single `List(limit=2)` call: if

the hash-suffixed key is already visible, the manifest hash can be derived from

the key name alone. Otherwise, Gitaly falls back to reading the generation

object itself.

Gitaly enumerates dependencies required for the MVCC snapshot through the `PATH`

chunk alone and does not have to interpret any other chunks. Prefetching is

idempotent because artifacts are immutable and content-addressed. Pinning with

external pointer file is created, and reads never advance the cache pointer.

{{% alert %}}

It is important that the manifest pointer is resolved to a specific version

exactly once for any RPC call and that all Git processes inherit that exact

version so that it is not possible to observe torn reads when there are

It is important that the current generation is resolved to a specific manifest

version exactly once for any RPC call and that all Git processes inherit that

exact version so that it is not possible to observe torn reads when there are

concurrent writers.

{{% /alert %}}

  alt rejected

    G-->>C: error

  else accepted

    G->>S: CAS manifest pointer

    alt CAS ok

    G->>S: conditional PUT generations/<current - 1>

    alt PUT ok

      S-->>G: ok

      G->>S: PUT generations/<current - 1>.<version>

      G-->>C: ok

    else CAS conflict

    else key already exists

      S-->>G: conflict

      G-->>C: retry/resolve/reject

    end

inspection may be to reach out to Rails' `/internal/allowed` checks, which would

then perform a set of reads against the new state. To allow these reads to be

distributed across nodes, Gitaly would have to upload artifacts to object

storage already before it updates the canonical and persistent manifest pointer.

Subsequent checks for this version should propagate `GIT_MVCC_MANIFEST` to point

to the proposed new version.

storage already before it writes the next generation object. Subsequent checks

for this version should propagate `GIT_MVCC_MANIFEST` to point to the proposed

new version.

Note that there is a tradeoff at play here:

once. From thereon, the temporary manifest shall be the single source of truth

for the manifest version for all subsequent Git processes in the mutating RPC.

Furthermore, the canonical manifest pointer must be updated _at most once_.

Furthermore, the next generation object must be written _at most once_.

Otherwise, the mutating RPC may result in torn writes.

{{% /alert %}}

When the new state was accepted and the artifacts have been uploaded, then

Gitaly performs a compare-and-swap operation of the manifest pointer. As there

can be multiple writers, this operation may fail due to a conflict. If so, there

are two scenarios:

Gitaly performs a conditional, create-only write of the next generation object.

As there can be multiple writers, this operation may fail because a concurrent

writer has already created that generation. If so, there are two scenarios:

- There can be a logical conflict because one reference is being updated to

  different versions. This will result in a reject and no update shall happen.

- There can be a conflict only in nature because the manifest pointer was

  changed, but none of the updates are conflicting. In this case, Gitaly will

- There can be a conflict only in nature because the current generation has

  advanced, but none of the updates are conflicting. In this case, Gitaly will

  try to resolve the conflict by doing a three-way merge of the changed

  references.

- Ours, which is the proposed update computed by the mutating RPC and that can

  be derived by reading the temporary manifest pointer once all writing Git

  commands have finished.

- Theirs, which is the current version that the canonical manifest pointer

  points to and that has been advanced by a concurrent writer.

- Theirs, which is the version named by the current generation and that has

  been advanced by a concurrent writer.

The merge then reads the references that have changed between these three

different versions and merges them. If any reference has received a conflicting

update then we have a logical conflict and reject the write. Otherwise, if no

reference has received multiple writes we can accept it.

reference has received multiple writes we can accept it and retry the

conditional write against the new current generation.

Note that there is no need for a three-way merge for objects. Instead, we can

treat them as conflict-free and always take the union of them.

Repositories that depend on these shared objects link to the object pool via the

Git alternates mechanism.

With the MVCC design, each Git repository has a manifest pointer that defines

With the MVCC design, each Git repository has a current generation that defines

current MVCC artifacts (reftables, packfiles, and manifest) in use. The manifest

specified by the repository's manifest pointer effectively allows Git to

named by the repository's current generation effectively allows Git to

construct an isolated snapshot/view of the repository. By associating

repositories with related histories (the upstream and its forks), it becomes

possible to deduplicate MVCC artifacts by using a shared pool of MVCC artifacts.

    r3[Repo C<br/>Fork]

    p1[Artifact pool 1]

  end

  r1 -->|Manifest pointer| p1

  r2 -->|Manifest pointer| p1

  r3 -->|Manifest pointer| p1

  r4 -->|Manifest pointer| p2

  r1 -->|Current manifest| p1

  r2 -->|Current manifest| p1

  r3 -->|Current manifest| p1

  r4 -->|Current manifest| p2

```

Note that creating a new repository sets up the repo in a new deduplication

network while creating a fork repository sets up the repo to join an existing

deduplication network. For this to work, an individual repository needs to know

the MVCC artifact pool it fetches from and also its manifest pointer. This

the MVCC artifact pool it fetches from and also its `generations/` prefix. This

information can be stored as a key-value mapping between the repository key and

its pool/manifest location. In a clustered setup, repositories using the same

its pool/generations location. In a clustered setup, repositories using the same

artifact pool will likely want to be routed to the same Gitaly node to enable

artifact deduplication in the local cache and reduce cold cache hits.

The new system does not come without its own risks:

- The added-latency-for-throughput tradeoff may prove unacceptable for some

  scenarios. A potential mitigation strategy is to cache resolved manifest

  pointers for a certain timeframe.

  scenarios. A potential mitigation strategy is to cache resolved generations

  for a certain timeframe.

- Cold-cache fill latency for repositories with many or large files is a

  concern. A potential mitigation strategy is eager cache warming.

- Compaction cadence and granularity must be balanced against

This means that it can be possible, under some scenarios, for more than one

Gitaly node to be issuing concurrent writes to a given repository. Conflicts may

arise when updating the manifest pointer, which will be dealt with separately.

arise when writing the next generation object, which will be dealt with

separately.

Given that local Gitaly disks are no longer the source of truth for durable

storage, these are acceptable tradeoffs to make for the sake of an architecture