Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 43,155
    • Issues 43,155
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,418
    • Merge requests 1,418
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #16267
Closed
Open
Created Oct 25, 2016 by Markus Kasten@markus-k

Keep latest artifacts for the last successful jobs

Problem to Solve

The expire_in function is great, but depending on how often your pipeline is built can be hard to tune. If you don't build frequently, setting a short expire_in can result in your most recent artifact being deleted. Setting it too long can keep unneeded copies around. Over time this can use up a lot of disk space. Most of the time, people just want to ensure they always at least have the most recent artifact, whenever that was built.

Proposal

Implement a default behavior for all projects to always keep the latest artifact for each successful job that was part of a successful pipeline (that produced artifacts) on any active (non-deleted) branch, merge request, or tag (i.e., the one downloaded via https://docs.gitlab.com/ce/user/project/pipelines/job_artifacts.html#downloading-the-latest-artifacts is exempt from any expire_in policy). In other words, never delete the most recent artifact (even if expired), which is same artifact as described at the docs link here:

image

Additionally:

  • Artifacts for deleted branches follow expire_in
  • All non-latest artifacts follow expire_in
  • Only the latest artifact for a job on a non-deleted branch is kept

This would allow for setting a relatively aggressive expiration policy, while remaining confident you always have your most recent artifact for any active branches.

Implementation notes

Because expire_in: could be set too short we would need to define a way to "lock" the latest artifacts to prevent deletion by ExpireBuildArtifactsWorker. For example add a new column job_artifacts.locked (false by default). Then when a pipeline completes we will lock the latest artifacts in a moving window: lock last ones and unlock anything from previous pipelines. Finally the ExpireBuildArtifactsWorker would need to be modified to filter out locked artifacts.

Another aspect to consider is to unlock artifacts when a branch is deleted or they would remain locked and never deleted.

Out of Scope for this MVC

Items not expected for this MVC issue:

  • ability to keep latest N count of artifacts per job
  • new checkbox in project's Settings > CI/CD > General pipelines for Keep latest job artifacts per ref
  • allow artifacts to live longer than N days for compliance reasons
  • aggressively remove artifacts which are not the latest even before they expire
Edited Jul 06, 2020 by Thao Yeager
Assignee
Assign to
Time tracking