2025-12-12: Jobs stuck in pending state with no corresponding entry in ci_pending_builds table
Jobs stuck in pending state with no corresponding entry in ci_pending_builds table (Severity 2)
Problem: CI jobs are stuck in the pending state with no corresponding entry in the ci_pending_builds table, preventing them from running.
Impact: Some CI jobs are stuck in the pending state and are never picked up by runners, causing pipelines to remain running until affected jobs are manually cancelled and retried. This blocks users from completing their workflows until they take manual action.
Causes: Jobs are being dropped from the pending queue while still in the 'created' state due to data consistency issues between the primary and replica databases. A race condition in the load balancer was fixed, but the main problem remains unidentified.
Response strategy: We have merged a mitigation to ensure CI builds are only dropped from the queue if they're confirmed to be in the 'pending' state. Release managers have prioritized deployment, and we are monitoring the rollout to assess its impact. Users can continue to cancel and retry affected jobs as a workaround. Further investigation and logging will continue if issues persist.
This ticket was created to track INC-6271, by incident.io