Option to only keep latest artifact for each branch.

Added ~19173 ~147988 ~14213 labels

@markus-k Thanks for the proposal! That seems like a sound strategy

This is pretty much what I'd like, too, except it would be great to give a number of old artifacts to keep. Something like keep_latest: 5 to keep the 5 most recent.

It sounds a really good proposal. Our product packaging downloads artifacts of multiple repositories and actually we have to set the expire_in to an unnecessary high value to be sure we always have artifacts available in some stale repositories (or we have to trigger build periodically on those repositories). With that keep_latest option we could reduce the disk cost and make a sysadmin happy !

mentioned in issue #27773 (closed)

the keep_artifact tag is just what I need

This would definitely help with encoring CI limits but still being useful: https://gitlab.com/gitlab-org/gitlab-ce/issues/23366

This would certainly be useful. See also Issue #19208 (moved)

Are there any workaround to emulate keep_latest artifact ?

mentioned in issue #31568 (closed)

mentioned in issue #3386 (closed)

I would like to see something like this:

artifacts:
  paths:
   - ${artifact_path}
  expire_in: 1d
  keep_last: 3

This would enforce an expiry of 1 day as long as there will always be 3 builds kept for backup/rollback/informational purposes.

any update on this getting implemented?

I strongly second this requirement.

mentioned in issue #19208 (moved)

@bikebilly Please can you take a look. This proposal is gaining some momentum now

This is a feature I've been looking for in Gitlab CI for quite some time as well!

Thank you everyone for the feedback, I'd like the keep_latest approach and the implications when used along with expire_in, even if I don't know how big is to access information about the number of artifacts that are from previous runs.

We also have some problems to address: how can you define if artifacts from previous pipelines counts? You may change the .gitlab-ci.yml, and this is quite hard to handle. You can change the job name, but want to consider it the same, or you can change the script of the job, and want to consider it a different one. How can you handle pipelines run on different branches? Is it latest three in the same branch, or in general?

Time-based expiration is relying only on the artifact itself, this approach relies on relationships and this introduces complexity and edge cases.

I'd like to know what the community thinkgs about my considerations.

Hi @bikebilly,

Cheers for getting back to us...

Will try to address points from my point of view... hopefully, it'll be helpful to you :)

Personally I'd be cool with having the change to .gitlab-ci.yml work retroactively, as one of the reasons I want this is to help save space as I've got some pipelines creating ridiculously huge artifacts.

I suppose one way to try to handle this differently would be to treat the artifact as an object and apply an expiration property to the artifact itself. This doesn't really address clearing that property if the .gitlab-ci.yml is changed after the fact though. Someone else might have a better solution for that!

As for changing of the job...

I guess the artifact should be an object with an 'expiration' property same as my previous suggestion..? I'm struggling to find another answer here.

For artifact behavior for specific branches. If the artifacts code block lives under a job definition then there are already ways to define that this is for a specific branch eg: for review branches in my pipeline I'm using: only: - /^review-.*$/

I guess the best suggestion I can offer that sort of covers your points is to handle the artifact as an object in code in the background of gitlab (not sure how difficult that would be as I haven't dug into it myself...) and then give users an interfact something like this maybe?:

# Inside a job context so that branches can be specified if the user wants to.
artifacts:
  paths:
   - ${artifact_path}
  expire_in: 1d
  keep_last: 3
  retroactive: True  # Default false

Hope this helps somewhat :)

mentioned in issue #36940 (closed)

marked this issue as related to #36940 (closed)

Hey @bikebilly has there been any updates in this space?

Had to go through and do a cleanup again recently on my gitlab instance!

@snayler0 unfortunately not yet. The only workaround I see now is to add a job in your pipeline that removes old artifacts using API, but I understand it is nothing more than a quick hack.

What about rollbacks?

Let's say I have a job that triggers during a Rollback which needs a previous job artifact, this would effectively fail if you only keep the latest artifact.

Does this even work now though?

@wakawaka54 For that reason I would prefer having a keep_last: <int> option to keep eg the last 3 artifacts that would be job-specific, rather than having a set expiry time.

In order to get this request some attention, this issue has been marked as a potentially interesting proposal as it meets the following criteria:

Labelled as a feature proposal
More than 10 upvotes
Unscheduled (not associated with a milestone)

Thanks for your proposal!

/cc @markglenfletcher

added auto updated potential proposal labels

added popular proposal label

mentioned in issue #24692 (moved)

This sounds like a great addition, would make our workflow much more streamlined. Would remove the need to fiddle with expiry times on a per project level.

mentioned in issue #43576 (moved)

Any news on this feature? In our Gitlab Enterprise edition we now have over 130GB of artifacts on 1 single branch, because it keeps all the older versions. Would be nice if we could only keep the latest.

That would greatly improve our workflow, the time it takes to back-up gitlab and disk space for storage.

One thing I like about the general spirit of the Gitlab-ci is its "stateless" approach. The fact that it is repeatable and the result is predictable.

So when we talk about keeping the "last few builds" I personally hope we talk about keeping the build artifacts associated with the last few commits of a branch.

I suppose that, from this point of view, we could continue to see the build artifacts somewhat stateless and decoupled - i.e. the granularity of a build artifacts would take into account a pipeline instance + job name + expiry / creation date but also a commit hash related to a branch.

Otherwise, if we only consider "the last few" from a time perspective, does it mean that triggering 3 times the same pipeline on the same branch would produce 3 different artifacts to keep? My personal answer to this would be no. Even if I could see why you might want that... it seems to me that this perspective brings a whole lot of complexity with it - i.e. the need for an artifacts manager and all the edge-cases to be handled "properly" (read: best-effort).

EDIT: Cool, doing this feature might actually help with the "rollback" feature not being very reliable right now, see gitlab-org/gitlab-ce#39539

This feature can help us too. As all mentioned, we too dont want to save all our artifacts but just few latest artifacts are needed. Thanks!

This would be nice when having e.g. both a backend and a frontend project, where building the backend requires the output from the frontend build.

However, another approach that might be even cleaner and would make this whole feature less important would be allowing to depend on a state from another project: Like this I could depend on a stage like 'frontend-repo/build' and if no artifacts from that stage exist, that project's CI would run to build them.

This would be really great :)

mentioned in issue #49677 (moved)

added devopsverify label

I have to add my voice to this one... I have Install package artifacts that needs to be available. Only the last successful build needs to be keept "forever"... So:

# Inside a job context so that branches can be specified if the user wants to.
artifacts:
  paths:
   - ${artifact_path}
  expire_in: 1d
  keep_last: 1

would work great for me (See https://gitlab.com/PRTG/Device-Templates/Aruba for an example)

@bikebilly Regarding the statelessness. Could you attach keep_last to each artifact, along with expire_in when you create the artifact? Then when you do the hourly cleanup check, you could check the keep_last of the latest artifact only from each branch for the value, and then use that as the basis for cleaning up the older artifacts on each branch, then fall back to expire_in for all artifacts which were greater than keep_last? I.e. skip the last keep_list artifacts before starting to check expire_in.

Or is this too simplistic?

added direction label

Customer interest --> https://gitlab.zendesk.com/agent/tickets/104091 (internal use)

added customer label

I guess the milestone "4-7" is an outdated info as Gitlab is currently in 11.4.x. Do you have a new goal?

@rriemann for me the milestone is called "Next 4-7 releases" which might indicate somewhen between 11.8 and 11.11 (counting from todays 11.4).

but then i don't know the exact semantics of the milestone.

added [deprecated] Accepting merge requests label

added 1 deleted label and removed 1 deleted label

added Category:Continuous Integration label

mentioned in issue #50939 (closed)

I wrote a tool that cleans up old images and artifacts: https://github.com/anordby/gitlab-registry-cicd-cleanup

My implementation:

uses Gitlab deployments API to check which images are in use and was in use, keeping the last n number of images per environment. That way you don't risk deleting images you are actively using or was using short time ago.
uses Gitlab jobs API for getting rid of jobs with artifacts that don't have expire set for artifacts. This takes up a lot of diskspace over time also. And it gives developers an incentive in using artifacts expiry to keep their old jobs around if they have any interest in that.
needs Gitlab new enough to have API v4.

Feel free to test, comment and supply patches to improve it :-)

Greetings from a snowy Oslo today, Anders.

marked #23180 (closed) as a duplicate of this issue

mentioned in issue haopoboy/ci-demo#1

removed milestone

removed direction label

Why was the milestone removed? This feature would be a great enrichment for us.

On 3/26/19 9:50 AM, Thomas Decaux wrote:

I am not sure this still make sense, because the new release API, I think all use cases are "covered" now:

artifact => share files between CI jobs, livecycle is CI / short

cache => share files between same CI job occurrences, livecycle is CI / long

release => store files permanently, livecycle is project / permanently

well... afaict, the "release assets" need a storage, which is currently not (directly) provided by the releases API.

i haven't found an easy to deploy method to automatically create releases from a tag in a CI-job yet (but that's probably a problem with documentation)

I couldn't find any feature that remove older artifact(s) when a new one was created. So this one should still be on the roadmap.

Two hundred upvotes is a strong indicator. There is a clear need and seemingly easy implementation. Please schedule it in some milestone.

A lot of our repositories see bursts of build activity, with relatively large artifacts. To avoid running out of space, we set pretty aggressive timeouts. Unfortunately, this often causes manual jobs to later fail. Having the option to retain the most recent artifacts would be extremely useful.

Earlier in the thread, there were questions about how to key the artifacts (e.g., job name, branch, etc). If that's the hang up, could it work like caching, where it can be customized?

keep_latest would be simple and would add a lot of value, especially when combined with latest artifact links.

We've migrated to GitLab from an in-house CI system (built before CI was mainstream), and it's been heartening to see the GitLab CI feature set grow to cover most of our requirements. FWIW, our internal system used a dynamic pruning algorithm to strike a balance between disk usage and maintaining a spread of representative builds.

We kept:

Every build on the current day
One per hour for previous days in the last week
One per day for builds older than a week, but less than 6 weeks
One per week for builds between 6w and 3m old
One per month for builds older than 3 months

So rather than a boolean option for keep_latest, or a numeric keep_last, it might be worth considering a syntax that allows alternative pruning algorithms in the future, e.g.

  artifacts:
    keep: latest

  artifacts:
    keep: last 42

  artifacts:
    keep: representatives

mentioned in issue #57635 (moved)

changed milestone to %12.7

changed milestone to %12.5

added direction label

Received a question about this from a customer today. Internal ZD link: https://gitlab.zendesk.com/agent/tickets/128174

added grouppipeline execution label

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

removed milestone

mentioned in issue gitlab#15361 (closed)

Option to only keep latest artifact for each branch.

Description

Proposal

Designs

Child items ...

Activity

Frequently Asked Questions

Option to only keep latest artifact for each branch.

Description

Proposal

Relates to

Activity

Frequently Asked Questions