Skip to content

Revert concurrent pipeline creation for pipeline schedules

What does this MR do?

We shipped https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28407 for mitigating a production incident gitlab-com/gl-infra/production#805 (closed). In that fix, we decided to create pipelines from pipeline schedules in parallel in order to avoid being killed by Sidekiq Memory Killer, however, due to the lock contention issue during the concurrent pipeline creation, it quickly eats up connection pool of PostgreSQL and we suspect this could result in gitlab-com/gl-infra/production#895 (closed).

This merge request reverts the concurrent pipeline creation in order to remove one of the potential cause of the new production incident. We do not have an evidence that this fix resolves the entire production fleet, however, it's worth trying.

This merge request reverts the concurrent pipeline creation, however, this would not reopen gitlab-com/gl-infra/production#805 (closed) because the current pipeline schedule architecture can resume from the checkpoint when PipelineSchedule died. This is because of next_run_at is calculated from the Cron worker's schedule. You can read more details in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28407.

Does this MR meet the acceptance criteria?

Conformity

Performance and testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda

Merge request reports