Prevent users from accidentally launching too many CI pipelines
This happened last month (gitlab-com/gl-infra/production#401 (closed)): the maintainer of Wireshark pushed up a new repo, and this kicked off hundreds of CI pipelines that put a significant strain on our file server nodes and runners. We had to manually kill the pipelines for GitLab.com to return to normal.
Again, today another team member accidentally pushed up all CE tags into the EE repo, and this wasn't the first time this has happened. We've had to manually kill off pipelines on dev.gitlab.org, but not before this has caused Sidekiq to get really busy and bring down the site.
I think we should consider:
- Limiting the number of pipelines created in the last X minutes (e.g. 10 in 1 minute).
- If it's a new repo, don't launch any CI pipelines after the first push to a repo (or at least one with a lot of tags/branches).
Marking this as a security issue because this could be used as a Denial of Service attack.
This also came up in a customer issue: https://gitlab.zendesk.com/agent/tickets/100092