Auto deploy pipelines failing because CNG deployment pipelines are still running
Over the last 24 hours we have seen all auto deploy pipelines starting to fail, due to the CNG build pipeline on dev.gitlab.org taking longer than usual.
If we look at the following pipelines (and their timings in minutes) we can see a significant increase at a point in time
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248803 (55 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248829 (54 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248866 (50 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248920 (83 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248941 (95 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248954 (76 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/248998 (85 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/249020 (86 minutes)
- https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/249025 (87 minutes)
Looking at the last pipeline that was "quick" and the first pipeline that was "slow" we can see it's the gitlab-rails-ee
job that has blown out the times
- https://dev.gitlab.org/gitlab/charts/components/images/-/jobs/13471635 (slow)
- https://dev.gitlab.org/gitlab/charts/components/images/-/jobs/13469047 (fast)
We need to determine why this change in timing has happened, if it's something we can remediate, or if we just have to adjust our job timings and live with it
Edited by Mayra Cabrera