gitlab-runner deletes stopped containers on upgrade, sometimes deleting all the things
Summary
The postinst script triggers a deletion of (some?) stopped docker containers. If the same apt operation also upgraded docker, that affects (nearly) all containers.
Steps to reproduce
- Install gitlab-runner and docker, both in a version which is upgradable
- Start a few containers
- Run
apt dist-upgrade
Actual behavior
Containers which were running before the upgrade get deleted.
Expected behavior
Containers should not be deleted because of an upgrade.
Relevant logs and/or screenshots
/var/log/apt/term.log
Log started: 2020-04-21 15:13:32
(Reading database ... 41237 files and directories currently installed.)
Preparing to unpack .../libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb ...
Unpacking libgnutls-dane0:amd64 (3.6.7-4+deb10u3) over (3.6.7-4+deb10u2) ...
Preparing to unpack .../libgnutls30_3.6.7-4+deb10u3_amd64.deb ...
Unpacking libgnutls30:amd64 (3.6.7-4+deb10u3) over (3.6.7-4+deb10u2) ...
Setting up libgnutls30:amd64 (3.6.7-4+deb10u3) ...
(Reading database ... 41237 files and directories currently installed.)
Preparing to unpack .../00-libssl1.1_1.1.1d-0+deb10u3_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1d-0+deb10u3) over (1.1.1d-0+deb10u2) ...
Preparing to unpack .../01-containerd.io_1.2.13-1_amd64.deb ...
Unpacking containerd.io (1.2.13-1) over (1.2.12-1) ...
Preparing to unpack .../02-curl_7.64.0-4+deb10u1_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u1) over (7.64.0-4) ...
Preparing to unpack .../03-libcurl4_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u1) over (7.64.0-4) ...
Preparing to unpack .../04-docker-ce-cli_5%3a19.03.8~3-0~debian-buster_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.8~3-0~debian-buster) over (5:19.03.6~3-0~debian-buster) ...
Preparing to unpack .../05-docker-ce_5%3a19.03.8~3-0~debian-buster_amd64.deb ...
Warning: Stopping docker.service, but it can still be activated by:
docker.socket
Unpacking docker-ce (5:19.03.8~3-0~debian-buster) over (5:19.03.6~3-0~debian-buster) ...
Preparing to unpack .../06-git-man_1%3a2.20.1-2+deb10u3_all.deb ...
Unpacking git-man (1:2.20.1-2+deb10u3) over (1:2.20.1-2+deb10u1) ...
Preparing to unpack .../07-libcurl3-gnutls_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) over (7.64.0-4) ...
Preparing to unpack .../08-git_1%3a2.20.1-2+deb10u3_amd64.deb ...
Unpacking git (1:2.20.1-2+deb10u3) over (1:2.20.1-2+deb10u1) ...
Preparing to unpack .../09-gitlab-runner_12.9.0_amd64.deb ...
Unpacking gitlab-runner (12.9.0) over (12.7.1) ...
Preparing to unpack .../10-libbluetooth3_5.50-1.2~deb10u1_amd64.deb ...
Unpacking libbluetooth3:amd64 (5.50-1.2~deb10u1) over (5.50-1) ...
Preparing to unpack .../11-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) over (63.1-6) ...
Preparing to unpack .../12-openssl_1.1.1d-0+deb10u3_amd64.deb ...
Unpacking openssl (1.1.1d-0+deb10u3) over (1.1.1d-0+deb10u2) ...
Setting up libgnutls-dane0:amd64 (3.6.7-4+deb10u3) ...
Setting up libssl1.1:amd64 (1.1.1d-0+deb10u3) ...
Setting up libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up containerd.io (1.2.13-1) ...
Installing new version of config file /etc/containerd/config.toml ...
Setting up docker-ce-cli (5:19.03.8~3-0~debian-buster) ...
Setting up libbluetooth3:amd64 (5.50-1.2~deb10u1) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u1) ...
Setting up git-man (1:2.20.1-2+deb10u3) ...
Setting up curl (7.64.0-4+deb10u1) ...
Setting up openssl (1.1.1d-0+deb10u3) ...
Setting up docker-ce (5:19.03.8~3-0~debian-buster) ...
Setting up git (1:2.20.1-2+deb10u3) ...
Setting up gitlab-runner (12.9.0) ...
GitLab Runner: detected user gitlab-runner
Runtime platform arch=amd64 os=linux pid=14694 revision=4c96e5ad version=12.9.0
gitlab-runner: Service is not running.
Runtime platform arch=amd64 os=linux pid=14704 revision=4c96e5ad version=12.9.0
gitlab-ci-multi-runner: Service is not running.
Runtime platform arch=amd64 os=linux pid=14737 revision=4c96e5ad version=12.9.0
Runtime platform arch=amd64 os=linux pid=14791 revision=4c96e5ad version=12.9.0
Clearing docker cache...
dc84e9e3434f
1ff5462f00bd
4660f67f02e3
e1724271f8f5
8cd702d9b7b5
a1b273517b69
494eea063c02
aa2983a82efb
25a3921a64ba
8d3b998809e3
Processing triggers for systemd (241-7~deb10u3) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
Log ended: 2020-04-21 15:15:01
Environment description
Debian GNU/Linux 10 (buster), Linux 4.19.0-8-amd64
$ apt policy gitlab-runner
gitlab-runner:
Installed: 12.9.0
Candidate: 12.9.0
Version table:
12.9.0+dfsg-1 -1
-1 http://debian.conova.eu/debian unstable/main amd64 Packages
-1 http://ftp.de.debian.org/debian unstable/main amd64 Packages
*** 12.9.0 1001
990 https://packages.gitlab.com/runner/gitlab-runner/debian buster/main amd64 Packages
100 /var/lib/dpkg/status
[…]
Used GitLab Runner version
root@makemake ~ › gitlab-runner --version
Version: 12.9.0
Git revision: 4c96e5ad
Git branch: 12-9-stable
GO version: go1.13.8
Built: 2020-03-20T13:01:56+0000
OS/Arch: linux/amd64
Possible fixes
Don’t delete containers on upgrade.
This is no fix, but here are the responsible code snippets:
/var/lib/dpkg/info/gitlab-runner.postinst
#!/bin/sh
set -e
case "$1" in
abort-upgrade|abort-remove|abort-deconfigure)
;;
configure)
/usr/share/gitlab-runner/post-install
/usr/share/gitlab-runner/clear-docker-cache || :
;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
exit 0
/usr/share/gitlab-runner/clear-docker-cache
#!/bin/sh
set -e
docker version >/dev/null 2>/dev/null
echo Clearing docker cache...
CONTAINERS=$(docker ps -a -q \
--filter=status=exited \
--filter=status=dead \
--filter=label=com.gitlab.gitlab-runner.type=cache)
if [ -n "${CONTAINERS}" ]; then
docker rm -v ${CONTAINERS}
fi
Edited by Jonas Schäfer