Use a per-VM git object pool for CI job fetches

What does this MR do?

As CI load increases due to agentic workflows (there are developers who are working on 90+ mrs concurrently via agents), we need to be smarter about how we run CI. Currently, every job does its own shallow clone. Though we have mechanisms through the stack to cache similar kinds of fetches, this is still wasteful. The gitlab project should find a way to clone in a central location and share the git data with every job.

flowchart TB
  subgraph vm["Job VM"]
    c1["Job 1 container"]
    c2["Job 2 container"]
    c3["Job 3 container"]
    pool["/cache/git-object-pools"]
  end
  c1 --- pool
  c2 --- pool
  c3 --- pool
  style vm stroke-dasharray: 5 5

This MR allows the runner helper container to seed the shared git object pool behind a CI variable so that multiple jobs can use those git objects through the git alternates mechanism.

Details

On runners that set GLCI_GIT_OBJECT_POOL_DIR, jobs borrow objects from a git object pool on the VM instead of fetching them all from Gitaly. Today that's the gitlab.com gitlab-org private shards p-s, p-m and p-l, turned on in the companion argocd MR, which point it at their per-project /cache volume. This merges inert: GLCI_GIT_OBJECT_POOL is "false" in this MR, so no job seeds a pool until it is turned on, which is a separate step after the runners are deployed.

• Seeding: a pre_get_sources_script step seeds $GLCI_GIT_OBJECT_POOL_DIR$CI_PROJECT_DIR/.git (for example /cache/git-object-pools/builds/gitlab-org/gitlab/.git) on the first job that finds no pool. It runs git -c transfer.bundleURI=true clone --bare --single-branch of the default branch, so the full clone comes from the bundle rather than Gitaly.
  • It clones into a temp path and renames it into place, sets gc.auto 0, and removes the job token before publishing.
  • It skips seeding when less than 20 GB is free, never runs for GIT_STRATEGY: none/empty, and never fails the job.
  • /cache is per project, and protected and unprotected refs each get their own volume and pool.
  • It only uses a GLCI_GIT_OBJECT_POOL_DIR under /cache/. A pool that is not a usable bare repository is removed and reseeded. If a get_sources attempt fails, the retry removes the pool and fetches normally, and the next job reseeds it, so a corrupted pool costs at most one failed attempt.
• Visibility: every job prints one Git object pool: line from the hook: borrowing from <pool> (<sha>), none seeded yet, not used by this job (GIT_STRATEGY: none/empty), or not configured on this runner. Only some runner managers set GLCI_GIT_OBJECT_POOL_DIR during the rollout, so this is how to tell from a job log whether it used a pool.
• Fetching: jobs keep the runner's normal git init + git fetch path and attach the pool as an alternate object store with GIT_ALTERNATE_OBJECT_DIRECTORIES: "${GLCI_GIT_OBJECT_POOL_DIR}${CI_PROJECT_DIR}/.git/objects". The runner only sets GLCI_GIT_OBJECT_POOL_DIR because it serves every gitlab-org project, so this project owns the pool name. Until a pool is seeded, the hook leaves an empty objects directory so git has nothing to complain about. Git finds objects already in the pool locally, so the fetch only transfers what the pool lacks.
• Runners without a pool: the runner expands variables by plain substitution, so an unset GLCI_GIT_OBJECT_POOL_DIR becomes empty and the value is the job's own .git/objects (for example /builds/gitlab-org/gitlab/.git/objects). Git ignores an alternate that is its own object directory, so it stays quiet in every stage, including jobs that skip get_sources. An earlier version pointed at a missing path there, and git printed error: unable to normalize alternate object path on every command, which broke danger-review (the git gem folds stderr into diff output) and the static-analysis jobs. Embedding CI_PROJECT_DIR is why the pool path is long.
• Seeding and borrowing stay in sync: a spec asserts that the variable git borrows through names the same pool the seeding hook creates, so a rename on one side alone fails CI.
• Native clone is avoided on purpose: FF_USE_GIT_NATIVE_CLONE resolves the mutable merge-request merge ref instead of refs/pipelines/<id>, which breaks pipelines whose target branch advances (gitlab-runner#39641 (closed)). We set neither it nor FF_USE_GIT_BUNDLE_URIS. Because bundle URIs are a clone-time mechanism, Gitaly also never serves a bundle for these jobs.
• clone-gitlab-repo sets GIT_ALTERNATE_OBJECT_DIRECTORIES: "". Its checkout becomes an artifact used on other VMs, so it must not depend on a per-VM pool. The job also asserts at runtime that it is not borrowing, so it fails loudly rather than publishing a .git artifact that depends on a VM-local pool.
• Turning it on: set GLCI_GIT_OBJECT_POOL to "true" once the runners are deployed. Setting it back to "false" stops seeding but not borrowing from a pool that already exists.

Companion MR: gitlab-com/gl-infra/argocd/apps!3948 (merged)

Relevant issues

Related to gitlab-com/gl-infra/production-engineering#29945

🤖 Generated with Claude Code