fix(daemon): Resume against the commit the mock actually serves
What this MR does and why?
A pipeline resumed after a daemon crash failed every job it had left to run, with an error that pointed at the user's repository rather than at glci.
glci does not serve the developer's repository as-is. PrepareGitRepoIn clones it --depth=1 and then rewrites the shallow clone into a synthetic orphan commit, and dirty mode — the default — stacks the uncommitted tree on top of that. The commit the mock server serves is therefore never the developer's HEAD. Preparation knows this: PreparePipeline overwrites its SHA from BareRepoHeadSHA once the bare repo exists, with a comment saying why, and BareRepoHeadSHA's own doc comment calls that value "the commit the repo serves: the daemon hands it to the runner as the SHA to check out".
The resume path did not. RepreparePipelineForResume rebuilt the git options by scanning CI_COMMIT_REF_NAME and CI_COMMIT_SHA out of a variable set re-derived from the working tree at resume time, under a comment reading "(same logic, no upload)" — but the logic was not the same, because the overwrite was missing. Every job dispatched fresh after a resume was told to check out a commit the served repository has never contained:
Checking out e5274aaf as detached HEAD (ref is main)...
fatal: unable to read tree (e5274aaf2ec946ad5aae439c232897716ba73584)
ERROR: Job failed: exit code 128Jobs already running when the daemon died were unaffected — monitorResumedJob builds no job spec, it waits on the surviving mock — so only the pending ones broke. With the default --git-strategy clone that is every resume of a pipeline with more than one stage.
The fix persists what preparation decided and restores it, which is the pattern already in that function three lines further down: SubmoduleRewrites and SubmoduleCredentialBase are restored from ExecutionState because "recomputing them would mean re-walking a bare repo this path deliberately never rebuilds". The served commit has the same property and was simply missed. ExecutionState gains ServedGitSHA and ServedGitRef.
Recomputing instead was not an option. unshallowToOrphan runs git commit-tree with a fixed author identity but no GIT_AUTHOR_DATE, so the commit is stamped at preparation time and a fresh preparation yields a different SHA; repeated runs only look stable because of a single-slot bare-repo cache that any other project's run evicts. Under --dirty the served commit is an overlay of a working tree that may have changed while the daemon was down, so it is unreproducible in principle. And re-uploading would move the served repo out from under the jobs monitorResumedJob is still watching.
Details worth a reviewer's attention:
- The ref is restored too, and #129 (closed) only names the commit.
Refwas re-derived fromCI_COMMIT_REF_NAMEin the same loop and drifts the same way — switch branches while the daemon is down and the fetch names a ref the--single-branchserved repo does not have. The two are restored as a pair, never independently: a ref from preparation beside a commit from the current working tree describes nothing that exists.RefTypeis not persisted; both construction sites on both paths hard-code"branch". - Under
git-strategy: remotea resumed pipeline now stays on the commit the original run pinned, rather than following a movedHEAD. This matches GitLab, which pins a retried pipeline throughrefs/pipelines/<id>—Ci::PersistentRefexists, in its own words, "to ensure runners can safely fetch source code even if force-push/source-branch-deletion happens". It is still a change #129 (closed) does not describe. CI_COMMIT_SHAis deliberately unchanged, on both paths. It still reports the developer's realHEAD; only the runner's checkout instruction was wrong. Note this means$CI_COMMIT_SHAandgit rev-parse HEADdisagree inside a job — a pre-existing divergence from GitLab that this MR neither introduces nor fixes.- A fallback is never written back as fact. When nothing usable was recorded — a pipeline prepared by an older glci, or one whose bare repo never got built — the resume falls back to
CI_COMMIT_SHAand warns.ExecutionParams.ServedGitKnownkeeps that guess out of the state file, so a second resume still warns instead of treating the guess as a record. - Both restored values are validated, and restored together or not at all. The commit must be a full lowercase hex object id, 40 characters or 64 under
--object-format=sha256; the ref must pass a documented subset ofgit check-ref-format. Both come off disk, so neither is trusted for reaching a git invocation inside the job container. The commit guard is the load-bearing one — gitlab-runner buildsgit checkout -f -q <sha>with no--separator and then slicesSha[0:8]without a length check — while the ref guard is defence in depth, sinceRefonly ever lands in an option-value position (--branch <ref>), apath.Join-anchored cache key, and log lines.
execStateVersion stays at 1. Both fields are additive and omitempty, ReadExecutionState only rejects state newer than it supports, and every optional field added so far went in without a bump. Bumping would make state written by this build unreadable — and so unresumable — to an older one.
The git-options rebuild moved into a pure resumeGitOptions helper. RepreparePipelineForResume needs Docker and cannot be unit-tested end to end; carving out a testable helper is what restoreSubmoduleRegistry already does for the submodule half of the same function.
Steps to reproduce
Needs a pipeline with a second stage still pending when the daemon dies, and the default clone strategy — git-strategy: none skips the checkout and hides this.
export GLCI_HOME=/tmp/glci-t129 # keep it short: a long path exceeds the
# unix socket's 104-char limit
mkdir -p /tmp/repro-129 && cd /tmp/repro-129
git init -q -b main .
echo "hello from the repo" > tracked-file.txt
cat > .gitlab-ci.yml <<'EOF'
stages: [slow, checkout]
slow-job:
stage: slow
image: alpine:3.20
script:
- echo "SLOW-JOB-STARTED"
- sleep 120
checkout-job:
stage: checkout
image: alpine:3.20
script:
- cat tracked-file.txt
EOF
git add -A && git commit -q -m ci
glci run -d
LOG=$(ls $GLCI_HOME/projects/*/pipelines/1/jobs/slow-job.log)
until grep -q SLOW-JOB-STARTED "$LOG"; do sleep 1; done
glci daemon stop --force # SIGKILL, so the mock and runner survive
glci daemon start # resume happens hereBefore: checkout-job fails in "Getting source from Git repository" with fatal: unable to read tree <your real HEAD>, while slow-job's log shows the runner checking out a different, synthetic commit. After: both jobs pass and checkout-job prints the file.
Automated as TestE2E_CrashRecovery_ResumeDispatchesPendingJob in the lifecycle tier (make test-integration-lifecycle). It was confirmed to fail against the unfixed code before being kept. The two existing recovery tests both pass git-strategy: none, which is exactly what dodges this, and both assert only that the pipeline finished — so a run that failed every pending job this way passed them.
Relevant issues and other links
Closes #129 (closed)