Windows killer copies osCmd with its sync.Once, so force kill closes the job object handle twice
Summary
newKiller on Windows copies the osCmd struct by value. The struct holds a sync.Once that guards closing the job object handle. After the copy, the killer and the original command each have their own Once. A force kill closes the job object handle through the copy. When the process exits, Wait closes the same handle value again through the original. The fix is to share one osCmd instead of copying it.
Why it matters: Windows can reuse a closed handle value, so the second close can close an unrelated handle in the runner.
Acceptance criteria
-
GOOS=windows go vet ./helpers/process/reports nocopylockswarning. - With
FF_USE_WINDOWS_JOB_OBJECTon, a force kill followed byWaitcloses the job object handle once. - A Windows test covers the force kill then
Waitsequence and fails on the current code.
Evidence
Versions
- Code read at commit
30f0b2097c8bbce086205e47607465ecb023c3c4onmain. Not tested on a released Runner version. - The
go vetwarning was also reproduced on the head of a branch that is one comment change ahead of that commit. - Not run on Windows: the double close and the handle reuse were read from the code, not observed. No Windows host was available.
Steps to reproduce
- Run
GOOS=windows go vet ./helpers/process/from the repository root. UseGOWORK=offif a parentgo.workinterferes. - Result:
helpers/process/killer_windows.go:27:11: literal copies lock value from *osCmd: ... contains sync.Once contains sync.noCopy.
The runtime sequence below was not run:
- Start a command with
FF_USE_WINDOWS_JOB_OBJECTon. - Cancel the job so
KillAndWaitruns and the graceful timeout expires. ForceKillcloses the job object handle through the killer's copy. The job kills the process.Waitreturns andcloseJobObjectcloses the same handle value through the original.
Logs and output
helpers/process/killer_windows.go:27:11: literal copies lock value from *osCmd: gitlab.com/gitlab-org/gitlab-runner/helpers/process.osCmd contains sync.Once contains sync.noCopyDetailed explanation (context for LLM agents and implementers)
Terms
- Job object: a Windows object that groups processes. With
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, closing its last handle kills the processes in it. - Killer: the
windowsKillerthatKillAndWaitbuilds to stop a process.
How it works today
osCmd.Startcreates the job object and stores the handle inc.jobObjectwhenUseWindowsJobObjectis set: job_windows.go#L25-48.KillAndWaitruns afterStart. It callsnewProcessKiller, thenTerminate, thenForceKillafter the graceful timeout: killer.go#L76-83.newKillercopies the command:cmd: *osCmd: killer_windows.go#L20-28. The copy has the handle value and a fresh, unusedOnce.ForceKillkills the process, then callscloseJobObjecton the copy: killer_windows.go#L45-55.closeJobObjectrunsCloseHandleinsideonce.Do: job_windows.go#L67-74.- The caller waits on
osCmd.Wait. After the process exits,WaitcallscloseJobObjecton the original, whoseOncewas never used: job_windows.go#L50-54.
What goes wrong, and where
Step 4 and step 6 each call CloseHandle on the same handle value, because the two Once values are separate. The second call fails with an invalid handle error that the code ignores, unless another component opened a handle in the meantime. Windows can give a closed handle value to the next CreateJobObject, file or pipe call. The second close then closes that other handle. Handle reuse was not observed.
History
Not determined. The same code is at the commit above, so it did not come from the merge request that removes FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY.
Related work
- #39289 (closed) : handle leak under the same feature flag (closed). It is where the close logic was added. Not verified that it introduced the copy.
Workaround
Do not set FF_USE_WINDOWS_JOB_OBJECT.
Proposed fix and fix options
- Option A: store the
*osCmdpointer inwindowsKillerand use it directly. OneOncethen guards both callers. This is the smallest change. Not tested. - Option B: move the
Oncebehind a pointer insideosCmd, so a copy shares it.
Tests to add
- File
helpers/process/killer_windows_test.goor a new*_windows_test.gofile. - Setup:
newOSCmdwithUseWindowsJobObject: true,Starta long-running command, callnewKiller(...).ForceKill(), callCreateJobObject(), then callWait. - Before the fix: if the new job object has the same handle value as the first,
WaitForSingleObjecton it fails afterWaitbecause it was closed. After the fix: it stays valid. - A compile-checked version of this test was written. It was not run.
Commands
GOOS=windows go vet ./helpers/process/GOOS=windows go test -c ./helpers/process/to compile the Windows tests on another OS.
Out of scope
- The removal of
FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY. This defect does not depend on it.