Inconsistencies between Private and Saas runners picking up Secure Test Project jobs
I'm seeing inconsistencies between Private and Saas runners picking up Secure Test Project jobs
Webgoat
Webgoat Package stage is failing with a docker error when picked up by the private shard, error:
cgroups: cgroup mountpoint does not exist: unknown
See for example https://gitlab.com/gitlab-org/security-products/tests/webgoat/-/jobs/4865341826
There is an open Docker issue that mentions that this is seen on some GitLab CI/CD jobs - https://github.com/docker/for-linux/issues/219 , with workarounds
Issue gitlab-runner#29132 relates to self hosted runner showing the same message
An example retry - original job https://gitlab.com/gitlab-org/security-products/tests/webgoat/-/jobs/4838010238 using runner
Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
on green-6.private.runners-manager.gitlab.com/gitlab.com/gitlab-org vHriyjxu, system ID: s_44fde8db605e
Retry passed https://gitlab.com/gitlab-org/security-products/tests/webgoat/-/jobs/4840475856 (see raw) using runner
Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
on blue-4.saas-linux-small-amd64.runners-manager.gitlab.com/default J2nyww-s, system ID: s_cf1798852952
Further example:
Fail with green runner https://gitlab.com/gitlab-org/security-products/tests/webgoat/-/jobs/4857223841
Pass with blue runner https://gitlab.com/gitlab-org/security-products/tests/webgoat/-/jobs/4867084277
Python Pip / IPTables
I'm seeing discrepancies between private/small on Python Pip offline tests
small
fails to run the iptables command
$ iptables -P INPUT DROP && iptables -P OUTPUT DROP
iptables v1.8.9 (nf_tables): Could not fetch rule set generation id: Invalid argument
and gives a false positive pass, as we are reaching out to GitLab to bring down gemnasium DB (which should fail the test)
private
runs the iptables command successfully which cuts connectivity, and fails (as it should do!) to contact GitLab
It explains why this test has appeared "flaky" as it also depends on the runner shard which picks it up.