gitlab-runner 11.9.0 stucks at fetching changes
We encountered a pipeline stuck since upgrade to 11.9.0. We run gitlab-runner under osx with shell executor, installed through homebrew.
This failure is constantly happening since we upgraded the runner.
Here is our debug log:
Running with gitlab-runner 11.9.0 (692ae235)
on macbook-runner eb913c5c
Using Shell executor...
+ set -eo pipefail
+ set +o noclobber
+ :
+ eval 'echo "Running on $(hostname)..."
'
+++ hostname
++ echo 'Running on company-build.local...'
Running on company-build.local...
EXPORTS PRINTED HERE
++ export GIT_LFS_SKIP_SMUDGE=1
++ GIT_LFS_SKIP_SMUDGE=1
++ mkdir -p /Users/company/builds/eb913c5c/0/example/product.tmp/git-template
++ git config -f /Users/company/builds/eb913c5c/0/example/product.tmp/git-template/config fetch.recurseSubmodules false
++ git config -f /Users/company/builds/eb913c5c/0/example/product.tmp/git-template/config http.https://gitlab.example.com.sslCAInfo /Users/company/builds/eb913c5c/0/example/product.tmp/CI_SERVER_TLS_CA_FILE
++ git init /Users/company/builds/eb913c5c/0/example/product --template /Users/company/builds/eb913c5c/0/example/product.tmp/git-template
Reinitialized existing Git repository in /Users/example/builds/eb913c5c/0/example/product/.git/
++ cd /Users/company/builds/eb913c5c/0/example/product
++ echo 'Fetching changes...'
Fetching changes...
++ git remote add origin https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.example.com/example/product.git
fatal: remote origin already exists.
++ git remote set-url origin https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.example.com/example/product.git
++ rm -f .git/index.lock
++ rm -f .git/shallow.lock
++ rm -f .git/HEAD.lock
++ rm -f .git/hooks/post-checkout
++ git clean -ffdx
++ git diff --no-ext-diff --quiet --exit-code
++ echo 'Clean repository'
Clean repository
++ git fetch origin --prune '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'
++ echo 'Checking out 97d18645 as feat/add-delete-behavior...'
Checking out 97d18645 as feat/add-delete-behavior...
++ git checkout -f -q 97d18645c0e2f64d254e8e36d2e5547a9fb3ce05
++ git-lfs version
++ git lfs pull
panic: sync: inconsistent mutex state
goroutine 13 [running]:
sync.(*Mutex).Lock(0xc8201024a0)
/usr/local/Cellar/go/1.5.1/libexec/src/sync/mutex.go:74 +0x16d
github.com/github/git-lfs/lfs.(*ScanRefsOptions).SetName(0xc8200fd500, 0xc8200c5140, 0x28, 0xc8200c5169, 0xa)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:91 +0x29
github.com/github/git-lfs/lfs.revListShas.func1(0xc82010b000, 0xc8200fd500, 0xc82001b980)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:292 +0x22c
created by github.com/github/git-lfs/lfs.revListShas
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:297 +0x306
goroutine 1 [runnable]:
syscall.Syscall(0x3, 0xd, 0xc820055300, 0x8, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.readlen(0xd, 0xc820055300, 0x8, 0x4, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/syscall/zsyscall_darwin_amd64.go:1392 +0x4c
syscall.forkExec(0xc820101900, 0x43, 0xc8201165a0, 0x3, 0x3, 0xc8200554d0, 0x6477, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/syscall/exec_unix.go:202 +0x567
syscall.StartProcess(0xc820101900, 0x43, 0xc8201165a0, 0x3, 0x3, 0xc8200554d0, 0x2, 0x4, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/syscall/exec_unix.go:240 +0x6b
os.startProcess(0xc820101900, 0x43, 0xc8201165a0, 0x3, 0x3, 0xc820055708, 0xc82010d600, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/os/exec_posix.go:45 +0x33c
os.StartProcess(0xc820101900, 0x43, 0xc8201165a0, 0x3, 0x3, 0xc820055708, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/os/doc.go:24 +0x69
os/exec.(*Cmd).Start(0xc820099540, 0x0, 0x0)
/usr/local/Cellar/go/1.5.1/libexec/src/os/exec/exec.go:328 +0x767
github.com/github/git-lfs/lfs.startCommand(0x490bd0, 0x3, 0xc820055988, 0x2, 0x2, 0xc82010b820, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:475 +0x325
github.com/github/git-lfs/lfs.catFileBatch(0xc82001bb60, 0xc82001bb60, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:401 +0xa6
github.com/github/git-lfs/lfs.ScanRefsToChan(0xc8200fc300, 0x28, 0x0, 0x0, 0xc8200fd500, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:146 +0x267
github.com/github/git-lfs/lfs.ScanRefs(0xc8200fc300, 0x28, 0x0, 0x0, 0xc8200fd500, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:107 +0x90
github.com/github/git-lfs/commands.pointersToFetchForRef(0xc8200fc300, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/commands/command_fetch.go:107 +0x13a
github.com/github/git-lfs/commands.fetchRefToChan(0xc8200fc300, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/commands/command_fetch.go:112 +0x60
github.com/github/git-lfs/commands.pullCommand(0x6bac00, 0x6e41d0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/commands/command_pull.go:45 +0x335
github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra.(*Command).execute(0x6bac00, 0x6e41d0, 0x0, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra/command.go:477 +0x403
github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra.(*Command).Execute(0x6bbc40, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra/command.go:551 +0x46a
github.com/github/git-lfs/commands.Run()
/Users/rick/go/src/github.com/github/git-lfs/commands/commands.go:88 +0x23
main.main()
/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:34 +0x12e
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1
goroutine 5 [syscall]:
os/signal.loop()
/usr/local/Cellar/go/1.5.1/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/Cellar/go/1.5.1/libexec/src/os/signal/signal_unix.go:28 +0x37
goroutine 7 [select, locked to thread]:
runtime.gopark(0x54e750, 0xc820032728, 0x4920a8, 0x6, 0x2d418, 0x2)
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820032728, 0x0, 0x18)
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820032728)
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/signal1_unix.go:227 +0x323
runtime.goexit()
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1
goroutine 8 [chan receive]:
main.main.func1(0xc82001af00, 0xc8200199b0)
/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:22 +0x6d
created by main.main
/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:32 +0x129
goroutine 14 [runnable]:
github.com/github/git-lfs/lfs.catFileBatchCheck.func1(0xc82010b820, 0xc82001bb60)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:365
created by github.com/github/git-lfs/lfs.catFileBatchCheck
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:384 +0x138
goroutine 15 [runnable]:
github.com/github/git-lfs/lfs.catFileBatchCheck.func2(0xc82001b980, 0xc82010b820)
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:386
created by github.com/github/git-lfs/lfs.catFileBatchCheck
/Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:391 +0x167
And the runner is stuck here until timeout occurs.
Useful information and fun facts
-
We never set
GIT_LFS_SKIP_SMUDGE
anywhere -
It occurs also on a brand new machine with this install process:
- Install osx xcode and command line tools
- install homebrew
home brew update && homebrew install gitlab-runner
- We don't install anything other through homebrew
-
gitlab-runner configure
-> we configure our runner setting shell as executor, our gitlab url and token, tags. Nothing other.
-
We NEVER installed git-lfs in the past (we don't even know what is it to be honest)
-
It worked properly under 11.8.0 and stopped working as soon as we updated to 11.9.0
-
We don't know
rick
. The stacktrace mention/usr/share/cellar/go
and/Users/rick
directories but we for sure don't have this on the machine. So I guess these one areinternal
to gitlab-runner homebrew installation? -
Our machine has 64Gb ram and 2Tb hard drive
-
Our repo is about 20mb in size