2026-05-07: Jobs duration is incorrect

Jobs duration is incorrect (Severity 2)

Problem: Job durations reported by GitLab were incorrect, with reported usage higher than the actual time jobs took to complete.

Impact: CI minutes usage was overstated for users, especially those running many jobs. This may have resulted in some users being billed for more CI minutes than they actually used. No customer complaints have been reported yet, but the issue could have become more visible as daily usage reports updated.

Causes: A regional stockout of N4d and T2D node types in GCP prevented pending pods from being scheduled, which delayed mitigation rollouts. Additionally, delays in the BuildTraceChunkFlushWorker process caused job completion times to be recorded later than when jobs actually finished, inflating reported durations.

Response strategy: We added N4 node types to the us-east1-c node pool to increase capacity and unblock mitigation rollouts. We also increased the urgent-ci-pipeline max replicas to 1200 to address queueing delays, and reverted a recent rerouting change that contributed to the issue. Manual checks show job duration discrepancies are now much reduced, though a small gap remains compared to baseline.

This ticket was created to track INC-9897, by incident.io 🔥