Add limit for daily scheduled pipelines for projects [RUN ALL RSPEC] [RUN AS-IF-FOSS] (!62826) · Merge requests · GitLab.org / GitLab

Furkan Ayhan requested to merge 323066-limit-for-scheduled-pipelines into master May 31, 2021

What does this MR do?

Background

Even if a schedule has * * * * *, our config make it run every 5 minutes at most */5 * * * *. So, a pipeline-schedule can create 288 pipelines per day at most.

So our goal is to limit that number for free-tier projects to prevent abuse.

As we limit all pipeline-schedules with the config */5 * * * *, let's have another config specifically for free-tier projects. 0 * * * *. With that, a pipeline-schedule with * * * * * will be able to run every hour (previously 5 minutes). It means, 24 pipelines per pipeline-schedule.

Now

This MR adds a limit for daily scheduled pipelines for projects.

Normally, pipeline schedules are triggered in a minimum frequency based on the pipeline_schedule_worker_cron value in the instance.

With this MR, we can adjust the minimum frequency based on the plan.

Related to #323066 (closed)

This is behind a feature flag ci_daily_limit_for_pipeline_schedules (#332333 (closed)).

Changes

Adding a ci_daily_pipeline_schedule_triggers integer column to the plan_limits table.
Inserting plan limits for ci_daily_pipeline_schedule_triggers, if GitLab.com.
- 24 (every hour) for free.
- 288 (every 5 minutes) for others.
Override the cron_worker_next_run_from method if the plan limit exists.
This will not affect self-hosted instances until they insert plan limits for pipeline_schedule_worker_cron.

Screenshots (strongly suggested)

In a frontend issue, we may add a warning/info banner when choosing the "custom" option here.

DB Review

Migration outputs (when not `GitLab.com`)

UP:

== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: migrating =====
-- add_column(:plan_limits, :ci_daily_pipeline_schedule_triggers, :integer, {:default=>0, :null=>false})
   -> 0.0110s
== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: migrated (0.0110s)

== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: migrating ====
== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: migrated (0.0000s)

DOWN:

== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: reverting ====
== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: reverted (0.0000s)

== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: reverting =====
-- remove_column(:plan_limits, :ci_daily_pipeline_schedule_triggers, :integer, {:default=>0, :null=>false})
   -> 0.0061s
== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: reverted (0.0087s)

Migration outputs (when `GitLab.com`)

UP:

== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: migrating =====
-- add_column(:plan_limits, :ci_daily_pipeline_schedule_triggers, :integer, {:default=>0, :null=>false})
   -> 0.0078s
== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: migrated (0.0079s)

== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: migrating ====
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("free")
   -> 0.0000s
-- quote(24)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '24' FROM plans WHERE name = 'free' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0041s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("bronze")
   -> 0.0000s
-- quote(288)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '288' FROM plans WHERE name = 'bronze' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0023s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("silver")
   -> 0.0000s
-- quote(288)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '288' FROM plans WHERE name = 'silver' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0007s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("gold")
   -> 0.0000s
-- quote(288)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '288' FROM plans WHERE name = 'gold' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0011s
== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: migrated (0.0340s)

DOWN:

== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: reverting ====
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("free")
   -> 0.0000s
-- quote(0)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '0' FROM plans WHERE name = 'free' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0034s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("bronze")
   -> 0.0000s
-- quote(0)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '0' FROM plans WHERE name = 'bronze' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0014s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("silver")
   -> 0.0000s
-- quote(0)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '0' FROM plans WHERE name = 'silver' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0014s
-- quote_column_name("ci_daily_pipeline_schedule_triggers")
   -> 0.0000s
-- quote("gold")
   -> 0.0000s
-- quote(0)
   -> 0.0000s
-- execute("INSERT INTO plan_limits (plan_id, \"ci_daily_pipeline_schedule_triggers\")\nSELECT id, '0' FROM plans WHERE name = 'gold' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"ci_daily_pipeline_schedule_triggers\" = EXCLUDED.\"ci_daily_pipeline_schedule_triggers\";\n")
   -> 0.0014s
== 20210526190553 InsertCiDailyPipelineScheduleTriggersPlanLimits: reverted (0.0088s)

== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: reverting =====
-- remove_column(:plan_limits, :ci_daily_pipeline_schedule_triggers, :integer, {:default=>0, :null=>false})
   -> 0.0040s
== 20210526190259 AddCiDailyPipelineScheduleTriggersToPlanLimits: reverted (0.0062s)

Does this MR meet the acceptance criteria?

Conformity

I have included a changelog entry, or it's not needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
I have tested this MR in all supported browsers, or it's not needed.
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Edited Jun 09, 2021 by Furkan Ayhan

Add limit for daily scheduled pipelines for projects [RUN ALL RSPEC] [RUN AS-IF-FOSS]