Skip to content

GitLab Runner doesn't clean up git lock files for submodules

Summary

GitLab Runner doesn't clean up lock files for submodules, causing submodule checkouts to fail in pipelines.

There's already logic for doing this for the main repository being checked out: https://gitlab.com/gitlab-org/gitlab-runner/blob/v10.6.0/shells/abstract.go#L101-105

However, there's nothing for traversing through the tree and purging those files from submodule checkouts.

Steps to reproduce

  1. Set up a repository that does a bunch of submodule checkouts
  2. Cancel midway through
  3. Start it again

Actual behavior

The checkouts fail on the lock files and the job fails.

Expected behavior

The checkouts should succeed (because the lock files are erased), and the job succeeds.

Relevant logs and/or screenshots

Fetching changes...
HEAD is now at 3dcda8d8 Update Doxyfile project version when using setversion.py
Checking out 1b02ea67 as doxy...
Updating/initializing submodules...
Synchronizing submodule url for 'lib/crc32c'
Synchronizing submodule url for 'lib/googletest'
Synchronizing submodule url for 'lib/json'
Synchronizing submodule url for 'lib/libiscsi'
Synchronizing submodule url for 'lib/libmicrohttpd'
Synchronizing submodule url for 'lib/libbaz'
Synchronizing submodule url for 'lib/libfoo'
Synchronizing submodule url for 'lib/libwin'

...

fatal: Unable to create '/builds/foo/liberty/.git/modules/lib/libwin/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Stopping at 'lib/libwin'; script returned non-zero status.

Environment description

We're using a custom local installation with Docker executors.

docker info output:

Containers: 58
 Running: 6
 Paused: 0
 Stopped: 52
Images: 16
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-112-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 60.95GiB
Name: ci-builder-2
ID: CKYP:UGDX:5VD6:DKWY:QVPJ:75E2:TU2H:6AWR:6U3B:VOL5:P3MP:LMWV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Used GitLab Runner version

Output of gitlab-runner --version:

Version:      10.6.0+datto.5.g1acc1d25
Git revision: 1acc1d25
Git branch:   v10.6.0-inc-timeouts-to-resolve-issue-2408-wip
GO version:   go1.9.4
Built:        2018-04-13T19:39:19+00:00
OS/Arch:      linux/amd64

The Git branch referenced is from here: https://gitlab.com/Conan_Kudo/gitlab-runner/tree/v10.6.0-inc-timeouts-to-resolve-issue-2408-wip

Edited by 🤖 GitLab Bot 🤖