Revert concurrent pipeline creation for pipeline schedules
What does this MR do?
We shipped https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28407 for mitigating a production incident gitlab-com/gl-infra/production#805 (closed). In that fix, we decided to create pipelines from pipeline schedules in parallel in order to avoid being killed by Sidekiq Memory Killer, however, due to the lock contention issue during the concurrent pipeline creation, it quickly eats up connection pool of PostgreSQL and we suspect this could result in gitlab-com/gl-infra/production#895 (closed).
This merge request reverts the concurrent pipeline creation in order to remove one of the potential cause of the new production incident. We do not have an evidence that this fix resolves the entire production fleet, however, it's worth trying.
This merge request reverts the concurrent pipeline creation, however, this would not reopen gitlab-com/gl-infra/production#805 (closed) because the current pipeline schedule architecture can resume from the checkpoint when PipelineSchedule
died. This is because of next_run_at
is calculated from the Cron worker's schedule. You can read more details in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28407.
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation created/updated or follow-up review issue created
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides - [-] Separation of EE specific content
Performance and testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team