GitLab Runner doesn't clean up git lock files for submodules
Summary
GitLab Runner doesn't clean up lock files for submodules, causing submodule checkouts to fail in pipelines.
There's already logic for doing this for the main repository being checked out: https://gitlab.com/gitlab-org/gitlab-runner/blob/v10.6.0/shells/abstract.go#L101-105
However, there's nothing for traversing through the tree and purging those files from submodule checkouts.
Steps to reproduce
- Set up a repository that does a bunch of submodule checkouts
- Cancel midway through
- Start it again
Actual behavior
The checkouts fail on the lock files and the job fails.
Expected behavior
The checkouts should succeed (because the lock files are erased), and the job succeeds.
Relevant logs and/or screenshots
Fetching changes...
HEAD is now at 3dcda8d8 Update Doxyfile project version when using setversion.py
Checking out 1b02ea67 as doxy...
Updating/initializing submodules...
Synchronizing submodule url for 'lib/crc32c'
Synchronizing submodule url for 'lib/googletest'
Synchronizing submodule url for 'lib/json'
Synchronizing submodule url for 'lib/libiscsi'
Synchronizing submodule url for 'lib/libmicrohttpd'
Synchronizing submodule url for 'lib/libbaz'
Synchronizing submodule url for 'lib/libfoo'
Synchronizing submodule url for 'lib/libwin'
...
fatal: Unable to create '/builds/foo/liberty/.git/modules/lib/libwin/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Stopping at 'lib/libwin'; script returned non-zero status.
Environment description
We're using a custom local installation with Docker executors.
docker info
output:
Containers: 58
Running: 6
Paused: 0
Stopped: 52
Images: 16
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-112-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 60.95GiB
Name: ci-builder-2
ID: CKYP:UGDX:5VD6:DKWY:QVPJ:75E2:TU2H:6AWR:6U3B:VOL5:P3MP:LMWV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Used GitLab Runner version
Output of gitlab-runner --version
:
Version: 10.6.0+datto.5.g1acc1d25
Git revision: 1acc1d25
Git branch: v10.6.0-inc-timeouts-to-resolve-issue-2408-wip
GO version: go1.9.4
Built: 2018-04-13T19:39:19+00:00
OS/Arch: linux/amd64
The Git branch referenced is from here: https://gitlab.com/Conan_Kudo/gitlab-runner/tree/v10.6.0-inc-timeouts-to-resolve-issue-2408-wip