Retry Windows and kubernetes integration jobs on script failure

What changes

The eight Windows integration test jobs and the kubernetes integration jobs are retried once when their script fails. Until now only runner failures were retried, twice, from the default: block. Those stay in the list. The retry limit is one number for all reasons, so a runner failure on these jobs is now retried once instead of twice. Windows unit and compile tests, lint and build jobs are not touched and still fail on the first script failure.

A shard that fails on setup, a flaky test or an OOMKilled helper now shows up as a retried job instead of a red pipeline, and in the merge train the car is not dropped. The same applies on main and tags, where these jobs also run.

Why

Between 2026-08-04 and 2026-09-03, 97 merge train pipelines failed. In 51 of them the only failed jobs were Windows integration or kubernetes integration jobs with a script failure. 62 of the 83 "pipeline did not succeed" removals were re-added with no new commit, after a median wait of 175 minutes and a new 97-minute pipeline. When someone retried one of these jobs by hand, the second attempt passed 62 of 70 times on Windows and 19 of 22 times on kubernetes. A retry costs one extra 40 to 55 minute Windows job or one 4 minute kubernetes job.

Over the same 30 days the Windows integration tests failed for a test-related reason on about 13 MRs, four of which look like real Windows problems, and every kubernetes integration failure was the whole suite hitting its 10-minute timeout. Extra runs on real failures are rare and cost one shard run.

The lower limit for runner failures has a cost: in the same 30 days, 7 runs of these jobs passed only on the third attempt after two runner failures. Those would now need a manual retry. That is small next to the 40 to 50 drops a month the script-failure retry removes.

Impact

A flaky shard becomes a 45-minute delay inside the same pipeline instead of a dropped car, a re-add the next morning and a second 97-minute pipeline. That is roughly half of today's merge train drops. The cost is one extra shard run each time a retry happens, and one fewer automatic retry for runner failures on these jobs.

Prediction: flaky merge train drops per completed merge train pipeline go from 0.30 to about 0.15, and the time from first "added to merge train" to merged, p90, goes from 84 hours to below 40.

How to verify after merge

Compare the 30 days before and after: merge train removals with reason "pipeline did not succeed" that were re-added without a new commit, divided by completed merge train pipelines, and the p90 of first "added to merge train" to merged. The number of automatic retries with failure_reason: script_failure on these jobs should be about the number of drops that no longer happen.

The Windows jobs allow exit code 99, which splitic uses when a test outside the flaky and quarantine lists fails. GitLab's retry looks at the failure reason, not at allow_failure, so an exit 99 is also retried. That is the wanted behaviour for a flaky test not yet on the list, and one extra shard run for a real failure. No Windows integration job finished with exit 99 in the 30 days measured.

Prepared by an agent from Igor's CI analysis of this project. The prediction is written down so it can be checked against the following weeks of pipelines.