gitlab-runner pipelines are unreliable

About 1 in 5 pipelines on gitlab-runner fails for reasons unrelated to the code being tested. Developers hit retry, it goes green, everyone moves on. This wastes CI capacity and developer time, and it trains people to ignore pipeline failures.

What's going wrong

Three months of pipeline data shows two distinct failure modes.

Flaky tests: 27 tests fail and then pass on retry with no code change. Race conditions in assertions, cancellation tests that don't account for all the ways a cancel can land, Windows process termination timing. These are real bugs in the tests, not in the code under test.

Infrastructure failures: Most of the problem isn't test bugs at all. Docker pull timeouts, preempted VMs, transient git clone failures. The jobs are fine, the environment isn't.

Proposed fix

For flaky tests: t.Skip the 27 identified tests. A flaky test doesn't catch regressions, it just randomly blocks merges. Each skip should get an owner and a timeline for a real fix. MR: !6575 (closed).

For infrastructure failures: add retry: 2 to affected jobs in .gitlab-ci.yml. This doesn't fix the underlying infra problems but stops them from blocking developers. MR: !6576 (closed).

What comes after

The skipped tests need to get fixed and re-enabled. The retries paper over real infra problems. This is triage, not a fix.

The analysis was done with gitlab-ci-jobs-export. flaky-tests.sh reproduces it end to end.