[CRITICAL] 2026-10-14: Downstream (child) pipelines not created on GitLab.com — bridge jobs stuck in pending since ~12:51 UTC

Customer Impact

Since ~12:51 UTC on 2026-10-14, GitLab.com does not create downstream (child) pipelines. Bridge (trigger) jobs stay in "pending" indefinitely with no error, so every CI stage that runs in a child pipeline (tests, builds, releases, deploys) is blocked. Multiple customers in different projects confirm the same symptom (see #21200), so any project using parent-child pipelines is likely affected.

Current Status

Ongoing as of 14:17 UTC (~1.5 h). No child pipeline has been created in the reporting project since 12:50:32 UTC, across 6 new pipelines on two branches. Two more customers reported the same symptom in #21200 at 14:07 and 14:16 UTC. Possibly related to the database / PGBouncer incidents #23189 (closed) and #23190; similar pending symptoms were also reported earlier in #21200. status.gitlab.com shows no CI/CD incident.

More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.

📝 Summary for CMOC notice / Exec summary:

  1. Customer Impact: Parent-child pipelines never start their child pipelines, so CI checks, builds and releases that depend on them are blocked for multiple customers.
  2. Service Impact: CI/CD pipeline creation (downstream pipelines), background processing (Sidekiq). Runners and regular jobs are not affected.
  3. Impact Duration: 2026-10-14 12:51 UTC - ongoing (90+ minutes at 14:17 UTC)
  4. Root cause: TBD. Possibly the main database / PGBouncer degradation (#23189 (closed), #23190), since child pipeline creation is a background job that writes to the main database.

Symptom

  • Bridge jobs (trigger: include: local, strategy: mirror) go to "pending" and stay there forever.
  • REST API: status: pending, downstream_pipeline: null, no failure_reason, queued_duration ~0.0001 s.
  • GraphQL: detailedStatus pending, hasDetails: false, downstreamPipeline: null; the parent pipeline's downstream.nodes is empty.
  • UI: no error or warning on the bridge jobs.
  • Jobs that depend on the bridges (e.g. gate:deploy) stay in "created" with runner: null, stuck: false, and runners: {online: true, available: true}.

Timeline (UTC, 2026-10-14)

  • 10:15 – pipeline 2946329725 (master): all bridges create their child pipelines within 1–3 s.
  • 12:11 – pipeline 2946759284 (master): all bridges create their child pipelines within 2–9 s.
  • 12:48:07 – child pipeline 2946898337 created (develop pipeline 2946892882).
  • 12:50:32 – child pipeline 2946907646 created. Last child pipeline created in the project.
  • 12:54 – pipeline 2946919757 (master, commit 7654921fc): bridges pending, no child pipelines; maintenance:panel (needs: []) pending from 12:54:42.
  • 13:02 – pipeline 2946940748 (master, re-run): same.
  • 13:10 – self-managed runner restarted; gitlab-runner verify passes; job pods run normally.
  • 13:11 – pipeline 2946967502 (master, re-run): same.
  • 13:15 – pipeline 2946981411 (master, re-run): same; bridge maintenance:panel (job 17172390588) pending from 13:15:57.
  • 13:17 – child pipelines in gitlab-org/gitlab are still being created.
  • 13:28 – pipeline 2947015915 (master, re-run): same.
  • 13:35 – pipeline 2947034702 (develop, empty commit 1420358e2): same; bridges pending from 13:36:33.
  • 13:42 / 13:47 – incidents #23189 (closed) (main DB disk degraded) and #23190 (PGBouncer saturation) opened.
  • 13:56 – pipeline 2947092109 (develop): same; maintenance:ci-cache (job 17173337014) pending from 13:56:49. Left running for inspection.
  • 14:07, 14:16 – two other customers report the same symptom in #21200.
  • 14:17 – still no child pipeline created.

What was ruled out

  • Configuration: the root config passes CI Lint on the current commit (22 jobs, no errors or warnings). All 9 child configs and their local includes parse. Between the last working master commit (99ae97190) and the current one (7654921fc), the only CI changes are comments in .gitlab-ci.yml and a few script lines in one child config (.gitlab/ci/maintenance.yml). Bridges whose child configs did not change hang the same way, an empty commit on develop reproduces it, and an invalid child config would fail the bridge instead of leaving it pending.
  • Trigger settings: trigger / strategy / include are unchanged since September; the same settings worked at 10:15 and 12:11.
  • Runners: both self-managed runners (Kubernetes and shell executors) are online and pick up regular jobs within seconds; GitLab itself reports them as online and available. A runner restart did not help. Bridges hang before any runner is involved.
  • Retries: six new pipelines on master and develop, all with the same result.
  • Rate limits: the Free-tier rate-limit preview window today starts at 15:00 UTC, after the problem started, and no HTTP 429 responses were seen.
  • Platform-wide outage: status.gitlab.com shows no CI/CD incident, and child pipelines in gitlab-org/gitlab were still being created at 13:17 UTC.

Details

📚 References and helpful links

Related:

  • #21200 – customer reports of pipelines stuck in pending, including today's child pipeline reports
  • #23190 – PGBouncer client connections saturation (Active)
  • #23189 (closed) – Compute Engine disk of the main DB degraded (Resolved)
  • #23179 (closed) – Sidekiq queueing on the low-urgency-ci-pipeline shard (yesterday, Resolved)

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.