Enable pipeline creation limits
In gitlab-foss#51401 (closed) we added support for job_activity_limit
as a limitation for the pipeline creation. The job_activity_limit
will drop a pipeline if the number of jobs in alive pipelines exceeds the specified limit.
Similar limits we have support for are pipeline_size_limit
and pipeline_activity_limit
.
Problem
Although the basic implementation for these limits is in place we haven't enabled them yet. In order to do that we would need to run the following in a production Rails console, given that they can only be set at DB level:
# iterate that for each plan
Plan::ALL_HOSTED_PLANS.each do |plan_name|
Plan.find_by_name(plan_name).update!(active_jobs_limit: 1000)
end
Or set specific (e.g. progressive) limits per plan.
However, there are a couple of issues to be addressed in order to enable these limits on production:
- it's currently not possible to set limits to
free
plan because it does not exist as a record in the DB and in the code anil
plan is consideredfree
. - we have not tracking/logging on when a limit is hit and drops a pipeline. Only
SELECT COUNT(*) FROM ci_pipelines WHERE failure_reason = 22;
Proposal
-
Add logging when a pipeline is dropped because of an exceeded limit (use Prometheus metrics if possible) !19155 (merged) -
Force free
plan to be a record in the DB and default to it whennamespace.actual_plan
isnil
!19033 (merged) -
Seed a Plan.create!(name: 'free', title: 'Free')
record on production !19033 (merged) -
Extract limits out of namespaces
table and into a dedicatedplan_limits
table !19438 (merged) -
Enable active_jobs_limit: 500
, especially those in the lower plans: E.g.free
,bronze
,silver
https://gitlab.com/gitlab-com/gl-infra/production/issues/1106
Edited by Fabio Pitino