fix(daemon): Resume against the commit the mock actually serves

What this MR does and why?

A pipeline resumed after a daemon crash failed every job it had left to run, with an error that pointed at the user's repository rather than at glci.

glci does not serve the developer's repository as-is. PrepareGitRepoIn clones it --depth=1 and then rewrites the shallow clone into a synthetic orphan commit, and dirty mode — the default — stacks the uncommitted tree on top of that. The commit the mock server serves is therefore never the developer's HEAD. Preparation knows this: PreparePipeline overwrites its SHA from BareRepoHeadSHA once the bare repo exists, with a comment saying why, and BareRepoHeadSHA's own doc comment calls that value "the commit the repo serves: the daemon hands it to the runner as the SHA to check out".

The resume path did not. RepreparePipelineForResume rebuilt the git options by scanning CI_COMMIT_REF_NAME and CI_COMMIT_SHA out of a variable set re-derived from the working tree at resume time, under a comment reading "(same logic, no upload)" — but the logic was not the same, because the overwrite was missing. Every job dispatched fresh after a resume was told to check out a commit the served repository has never contained:

Checking out e5274aaf as detached HEAD (ref is main)...
fatal: unable to read tree (e5274aaf2ec946ad5aae439c232897716ba73584)
ERROR: Job failed: exit code 128

Jobs already running when the daemon died were unaffected — monitorResumedJob builds no job spec, it waits on the surviving mock — so only the pending ones broke. With the default --git-strategy clone that is every resume of a pipeline with more than one stage.

The fix persists what preparation decided and restores it, which is the pattern already in that function three lines further down: SubmoduleRewrites and SubmoduleCredentialBase are restored from ExecutionState because "recomputing them would mean re-walking a bare repo this path deliberately never rebuilds". The served commit has the same property and was simply missed. ExecutionState gains ServedGitSHA and ServedGitRef.

Recomputing instead was not an option. unshallowToOrphan runs git commit-tree with a fixed author identity but no GIT_AUTHOR_DATE, so the commit is stamped at preparation time and a fresh preparation yields a different SHA; repeated runs only look stable because of a single-slot bare-repo cache that any other project's run evicts. Under --dirty the served commit is an overlay of a working tree that may have changed while the daemon was down, so it is unreproducible in principle. And re-uploading would move the served repo out from under the jobs monitorResumedJob is still watching.

Details worth a reviewer's attention:

  • The ref is restored too, and #129 (closed) only names the commit. Ref was re-derived from CI_COMMIT_REF_NAME in the same loop and drifts the same way — switch branches while the daemon is down and the fetch names a ref the --single-branch served repo does not have. The two are restored as a pair, never independently: a ref from preparation beside a commit from the current working tree describes nothing that exists. RefType is not persisted; both construction sites on both paths hard-code "branch".
  • Under git-strategy: remote a resumed pipeline now stays on the commit the original run pinned, rather than following a moved HEAD. This matches GitLab, which pins a retried pipeline through refs/pipelines/<id>Ci::PersistentRef exists, in its own words, "to ensure runners can safely fetch source code even if force-push/source-branch-deletion happens". It is still a change #129 (closed) does not describe.
  • CI_COMMIT_SHA is deliberately unchanged, on both paths. It still reports the developer's real HEAD; only the runner's checkout instruction was wrong. Note this means $CI_COMMIT_SHA and git rev-parse HEAD disagree inside a job — a pre-existing divergence from GitLab that this MR neither introduces nor fixes.
  • A fallback is never written back as fact. When nothing usable was recorded — a pipeline prepared by an older glci, or one whose bare repo never got built — the resume falls back to CI_COMMIT_SHA and warns. ExecutionParams.ServedGitKnown keeps that guess out of the state file, so a second resume still warns instead of treating the guess as a record.
  • Both restored values are validated, and restored together or not at all. The commit must be a full lowercase hex object id, 40 characters or 64 under --object-format=sha256; the ref must pass a documented subset of git check-ref-format. Both come off disk, so neither is trusted for reaching a git invocation inside the job container. The commit guard is the load-bearing one — gitlab-runner builds git checkout -f -q <sha> with no -- separator and then slices Sha[0:8] without a length check — while the ref guard is defence in depth, since Ref only ever lands in an option-value position (--branch <ref>), a path.Join-anchored cache key, and log lines.

execStateVersion stays at 1. Both fields are additive and omitempty, ReadExecutionState only rejects state newer than it supports, and every optional field added so far went in without a bump. Bumping would make state written by this build unreadable — and so unresumable — to an older one.

The git-options rebuild moved into a pure resumeGitOptions helper. RepreparePipelineForResume needs Docker and cannot be unit-tested end to end; carving out a testable helper is what restoreSubmoduleRegistry already does for the submodule half of the same function.

Steps to reproduce

Needs a pipeline with a second stage still pending when the daemon dies, and the default clone strategy — git-strategy: none skips the checkout and hides this.

export GLCI_HOME=/tmp/glci-t129        # keep it short: a long path exceeds the
                                       # unix socket's 104-char limit
mkdir -p /tmp/repro-129 && cd /tmp/repro-129
git init -q -b main .
echo "hello from the repo" > tracked-file.txt

cat > .gitlab-ci.yml <<'EOF'
stages: [slow, checkout]

slow-job:
  stage: slow
  image: alpine:3.20
  script:
    - echo "SLOW-JOB-STARTED"
    - sleep 120

checkout-job:
  stage: checkout
  image: alpine:3.20
  script:
    - cat tracked-file.txt
EOF
git add -A && git commit -q -m ci

glci run -d
LOG=$(ls $GLCI_HOME/projects/*/pipelines/1/jobs/slow-job.log)
until grep -q SLOW-JOB-STARTED "$LOG"; do sleep 1; done

glci daemon stop --force               # SIGKILL, so the mock and runner survive
glci daemon start                      # resume happens here

Before: checkout-job fails in "Getting source from Git repository" with fatal: unable to read tree <your real HEAD>, while slow-job's log shows the runner checking out a different, synthetic commit. After: both jobs pass and checkout-job prints the file.

Automated as TestE2E_CrashRecovery_ResumeDispatchesPendingJob in the lifecycle tier (make test-integration-lifecycle). It was confirmed to fail against the unfixed code before being kept. The two existing recovery tests both pass git-strategy: none, which is exactly what dodges this, and both assert only that the pipeline finished — so a run that failed every pending job this way passed them.

Closes #129 (closed)

Edited by Paweł Farys

Merge request reports

Loading
Loading