Windows integration test jobs fail due to fatal splitic timing-fetch errors
Windows integration test jobs in this pipeline fail frequently. One confirmed root cause has been filed upstream against splitic:
gitlab-org/ci-cd/runner-tools/splitic#1 (closed)
Summary: before running any tests, splitic test fetches per-test timing data from a previous pipeline's test_report API endpoint to balance parallel shards. Any error from that fetch other than "no timing data available" is currently treated as fatal (os.Exit(1)), aborting the whole job before a single test runs. A transient timeout hitting gitlab.com/api/v4 (10s ResponseHeaderTimeout) is enough to kill the job with no test output and no artifacts produced.
Observed here: https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/15163231209
$ splitic test -flaky ... -tags integration ./... -- -timeout 55m
error collecting timing information: performing timing request:
Get "https://gitlab.com/api/v4/projects/250833/pipelines/2648455886/test_report":
http2: timeout awaiting response headersNote: this is not the only cause of Windows integration test failures observed. Other distinct signatures seen in recent failed jobs:
- Job terminates silently right after the
choco install zstandardstep, beforesplitic testeven runs, with no captured error. error running tests: truncated test output: panic occurred?— a panic during test execution.
Those may warrant separate investigation/issues. This issue tracks the splitic timing-fetch cause specifically and should be closed once splitic#1 is fixed and rolled out to the version this pipeline uses (go install gitlab.com/ajwalker/splitic@latest).