Remove CI builds/jobs older than X years

Problem to Solve

Allow users to configure removal of older builds/jobs for their namespace or instance.

Questions to Answer

1. What would it look like to remove CI builds (and all its artifacts) older than X years?
2. How could we enable this on gitlab.com? What comms would we need for our users? Would this be a breaking change, and therefore cannot enable til x.0?
3. What if this could be configurable for Self-Managed and Dedicated customers who manage their own data?

Implementation

See linked MRs

Follow-up Tasks

• Ensure the setting can only be viewable by those who have the Owner role of a project
• Assign Delete old pipelines (!171142 - merged) for UX Review
• Create a feature flag rollout issue (can we roll out to GitLab team members first?)
• Merge Add docs for automatic pipeline cleanup (!174963 - merged)
• Add a to the Settings and a link to the docs once ready
• Prepare a Release Post linking to automatically delete old pipelines (#338480 - closed) - also confirm if this issue is a dupe of that one

added gitlab-org#11878 as parent epic

changed milestone to %Next 1-3 releases

added Category:Continuous Integration (CI) Scaling Engineering Time backend database devopsverify groupci platform maintenanceremoval sectionci severity1 typemaintenance labels

changed title from Feature request: Remove CI builds/jobs older than X years to Remove CI builds/jobs older than X years

changed the description

cc: @tianwenchen @mbobin

mentioned in issue gitlab-org/database-team/team-tasks#468 (closed)

@cheryl.li I think this should be at the pipeline level. #271614 (closed) has more details, but primary it's because builds could be retried at a later date and if we remove based on the creation date we would end up with partial pipelines.

@mbobin Would this be difficult to implement such a feature where you delete pipelines per namespace older than "X"? As a first iteration, could it be an experimental feature we allow users to enable such a setting in the UI? Or the "soon to be deleted" (expired) pipelines sit in a trash bin for Y days? WDYT? automatically delete old pipelines (#338480 - closed) seems related.

@cheryl.li I think the most expensive part to implement is finding the pipelines that we want to remove, but as an experimental opt-in feature I think we could do something with a weight 3 because that would give us less projects to iterate through in the begging:

add `delete_pipelines_in_seconds` column to `project_ci_cd_settings`

# Add a new cron worker that does this:
ProjectCiCdSetting.where.not(delete_pipelines_in_seconds: nil).find_each do |settings|
  Ci::Pipeline.where(project_id: settings.project_id).where('created_at < ?', delete_pipelines_in_seconds.seconds.ago).find_each do |pipeline| # or pluck a limited number of ids to ensure fair scheduling
    # this would be better in a new sidekiq worker    
    # the service needs a `current_user` to check for permissions and log a pipeline deleted event, maybe we can use the pipeline's author as the user that's deleting the record?
    ::Ci::DestroyPipelineService.new(pipeline.project, pipeline.user).execute(pipeline) 
  end
end

I think something like this could handle a few thousand projects?

Taking a stab at it in !171142 (merged)

FYI @drew as we discussed this in our 1-1.

mentioned in merge request !171142 (merged)

mentioned in issue #503153 (closed)

changed milestone to %17.6

added workflowin review label

changed milestone to %17.7

added missed:17.6 label

mentioned in merge request !173512 (merged)

mentioned in merge request !173982 (merged)

mentioned in merge request !174963 (merged)

added workflowverification label and removed workflowin review label