Tracking all open infradev issues for ~"group::pipeline execution" across `gitlab-org/gitlab` and `gitlab-com/gl-infra/production-engineering`. ## Overdue ~"group::pipeline execution" ~"infradev" — past due date ```glql display: table title: Overdue Infradev Issues (gitlab-org/gitlab) fields: title, assignee, labels("severity::1", "severity::2", "severity::3") AS "Severity", milestone, status, due limit: 20 query: project = "gitlab-org/gitlab" AND type = Issue AND state = opened AND label = "group::pipeline execution" AND label = "infradev" AND due < today() sort: due asc ``` ```glql display: table title: Overdue Infradev Issues (production-engineering) fields: title, assignee, labels("severity::1", "severity::2", "severity::3") AS "Severity", milestone, status, due limit: 20 query: project = "gitlab-com/gl-infra/production-engineering" AND type = Issue AND state = opened AND label = "group::pipeline execution" AND label = "infradev" AND due < today() sort: due asc ``` ## Upcoming / Not Yet Due ~"group::pipeline execution" ~"infradev" — open, with a future due date ```glql display: table title: Upcoming Infradev Issues (gitlab-org/gitlab) fields: title, assignee, labels("severity::1", "severity::2", "severity::3") AS "Severity", milestone, status, due limit: 20 query: project = "gitlab-org/gitlab" AND type = Issue AND state = opened AND label = "group::pipeline execution" AND label = "infradev" AND due >= today() sort: due asc ``` ```glql display: table title: Upcoming Infradev Issues (production-engineering) fields: title, assignee, labels("severity::1", "severity::2", "severity::3") AS "Severity", milestone, status, due limit: 20 query: project = "gitlab-com/gl-infra/production-engineering" AND type = Issue AND state = opened AND label = "group::pipeline execution" AND label = "infradev" AND due >= today() sort: due asc ``` ## Status Summary Status snapshot for each open infradev issue, ordered roughly from most-overdue to newest. Flagging blockers and "effectively done" items so they can be moved off the list. Last refreshed 2026-06-16. ### Overdue #### [#490968 Database contention in ci_namespace_monthly_usages updates for large, nested top-level group](https://gitlab.com/gitlab-org/gitlab/-/work_items/490968) — S2/P2, "Next 1-3 releases", due 2026-03-10 **In dev** with @srajadas, parented under epic [&17996 Ci Minute Bugs and tech debt](https://gitlab.com/groups/gitlab-org/-/work_items/17996). Long-running effort to shard `ci_namespace_monthly_usages` and aggregate reads via Redis. Multiple MRs merged ([!205603](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/205603), [!225852](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/225852), [!229811](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/229811)), with [!224699 Shard namespace usages](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/224699) still in draft. Has been missed across 10+ milestones (18.2 through 18.11) — work is progressing but slipping. #### [#582085 Self-heal pipelines stuck with an executing status with no executing builds](https://gitlab.com/gitlab-org/gitlab/-/work_items/582085) — S2/P1, 19.2, due 2026-04-02 **In dev**, assigned to @tigerwnz, parented under epic [&12910 Support Pipeline Management at scale](https://gitlab.com/groups/gitlab-org/-/work_items/12910). No longer `workflow::blocked` — block has been cleared. Core MR [!231261 Add worker to self-heal stuck pipelines](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/231261) is still open; supporting index MRs ([!232053](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/232053), [!230652](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230652)) are merged. Milestone slipped from 19.0 → 19.1 → now 19.2; needs the main MR landed. #### [#589486 Reduce idle-in-transaction time of the Ci::Minutes::UpdateProjectAndNamespaceUsageWorker](https://gitlab.com/gitlab-org/gitlab/-/work_items/589486) — S2/P3, "Next 1-3 releases", due 2026-03-11 (confidential) **In dev** with @srajadas, parented under epic [&20903 PgBouncer idle-in-transaction](https://gitlab.com/groups/gitlab-org/-/work_items/20903). Shares MRs with [#490968](https://gitlab.com/gitlab-org/gitlab/-/work_items/490968) including [!224699 Shard namespace usages](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/224699). Recent work has expanded into fixing the empty projects quota table via new GraphQL fields ([!239504](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239504) merged, [!239495](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239495) and [!239510](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239510) open). Progressing alongside the parent compute-minutes work. #### [#592819 Pipeline durations must be an SLI](https://gitlab.com/gitlab-org/gitlab/-/work_items/592819) — S2/P2, Backlog, due 2026-04-06 **In dev** with @hfyngvason. Substantial progress: several SLI instrumentation MRs merged ([!229670](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/229670), [!230396](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230396), runbooks [!10360](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/10360), handbook scope MRs [!19378](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/19378), [!19379](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/19379)) with one SLI MR still open ([!229728](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/229728)). Note the processing-delay SLI MR [!239278](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239278) was closed. Still in Backlog milestone despite being in dev — milestone should be set. #### [#594031 CI/CD compute minute billing inflated by BuildTraceChunkFlushWorker delays and 202 retry loop](https://gitlab.com/gitlab-org/gitlab/-/work_items/594031) — S1/P1, "Next 1-3 releases", due 2026-05-14 (confidential) **In dev** with @hfyngvason. Primary fix [!239131 Anchor build finished_at to pending_state.created_at](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239131) is now **merged**, gated behind feature flag [#601976](https://gitlab.com/gitlab-org/gitlab/-/work_items/601976) (`ci_anchor_finished_at_to_pending_state`). Partition-pruning fixes from the incident also merged. Next step is flag rollout; the longer-term sidekiq-amplification idea (idempotent `accept_build_state!` + cron over `BuildPendingState`) is still under discussion. #### [#597558 Self-hosted docker upgrade from 18.10 to 18.11 fails (missing index_deployments_on_id_and_status_and_created_at)](https://gitlab.com/gitlab-org/gitlab/-/work_items/597558) — S1/P1, 19.1, due 2026-06-02 **Verification** with @mfanGitLab. Fix + backports all merged ([!237966](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/237966), [!237974](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/237974), [!237997](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/237997)). Milestone now set to 19.1. **Effectively done** — only verification remains (re-evaluate AMER-2 tenant upgrades after APAC). Should close shortly. ### Not Yet Due #### [#29150 Reinstate the merge request 'Create pipeline creation requests at the start of PostReceive (FF)'](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29150) — S3, no milestone, due 2026-07-30 **In dev** with @hfyngvason. Follow-up from INC-10762 where the original MR ([!232296](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/232296)) was reverted ([!239194](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239194)). Reinstatement draft open ([!239293](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239293)) gated behind feature flag `track_ref_pipeline_creation`. On track. ### Themes / Cross-cutting observations - **Compute-minutes billing accuracy** remains the dominant theme: [#490968](https://gitlab.com/gitlab-org/gitlab/-/work_items/490968), [#589486](https://gitlab.com/gitlab-org/gitlab/-/work_items/589486), and [#594031](https://gitlab.com/gitlab-org/gitlab/-/work_items/594031) all relate to over-billing or namespace usage contention. - **Workflow hygiene**: [#592819](https://gitlab.com/gitlab-org/gitlab/-/work_items/592819) has merged MRs but is still in Backlog milestone; [#597765](https://gitlab.com/gitlab-org/gitlab/-/work_items/597765) has merged patches but is still in refinement with no milestone. - **Lacks a milestone** despite being open and overdue: [#597765](https://gitlab.com/gitlab-org/gitlab/-/work_items/597765). - **Effectively done pending verification**: [#597558](https://gitlab.com/gitlab-org/gitlab/-/work_items/597558) (verification) and [#594031](https://gitlab.com/gitlab-org/gitlab/-/work_items/594031) (merged, awaiting flag rollout).