<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> ### Release notes Setting an aggressive expiration policy for artifacts is one strategy for managing storage consumption, but sometimes you need the ability to reduce storage right away. While previously you may have relied on a script to automate the tedious task of calling the API to delete an artifact one by one, now you can use a new endpoint to bulk delete job artifacts. https://docs.gitlab.com/ee/api/job_artifacts.html#delete-artifacts ### Problem to solve Today there is no method to clean up old artifacts in bulk, short of writing your own script using the API. This present a challenge, as Artifacts are a major driver of storage consumption and many users today do not have any expiration policy set (currently by default artifacts never expire). Even future work to set a default of one-year expiration may not clean up old artifacts quickly enough to reduce usage below limits defined for storage quotas. ### Intended users * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) ### User experience goal <!-- What is the single user experience workflow this problem addresses? For example, "The user should be able to use the UI/API/.gitlab-ci.yml with GitLab to <perform a specific task>" https://about.gitlab.com/handbook/engineering/ux/ux-research-training/user-story-mapping/ --> Give users an API to deleting artifacts in bulk without them having to write their own scripts in the API or deleting these by hand one by one. ### Proposal <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> Create an API to bulk delete artifacts per project. The scope of work for this issue is to delete artifacts for a given project via an API endpoint. #### Out of scope (follow-ups) Add parameters to bulk delete endpoints: * `delete_all`: should allow for the deletion of non-erasable artifacts(trace files) * `delete_tags`: delete artifacts also for tags, * `keep_n`: keep N last artifacts for succesful pipelines on each branch, * `delete_orphaned`: delete all orphaned branch and tags (no longer present in git repo), * `older_than`: delete older than Delete all of the artifacts for a given group. #### Technical approach Extract the part which [removes](https://gitlab.com/gitlab-org/gitlab/-/blob/9fc33b5917423111cbff1d0c25544cbbabbb1ce3/app/services/ci/destroy_expired_job_artifacts_service.rb#L71-80) expired artifacts into a new service that accepts an array of job artifacts. This service could be reused for this API. Note we should remove only [erasable artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/9fc33b5917423111cbff1d0c25544cbbabbb1ce3/app/models/ci/job_artifact.rb#L164-169) by default. ### Further details We are working to apply a default expiration, but to avoid deleting important customer data we have a long grace period where users can retain their artifacts: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10177, currently artifacts will not expire sooner than 1 year. The support team has workaround process to do something similar, documented at: https://docs.gitlab.com/ee/administration/job_artifacts.html#delete-job-artifacts-from-jobs-completed-before-a-specific-date ### Permissions and Security Who can use this API for a project is critical. Similar to the existing [CI/CD permissions](https://docs.gitlab.com/ee/user/permissions.html#gitlab-cicd-permissions) related to deleting a project use of this API should be limited to maintainers and owners. <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? Consider adding checkboxes and expectations of users with certain levels of membership https://docs.gitlab.com/ee/user/permissions.html * [ ] Add expected impact to members with no access (0) * [ ] Add expected impact to Guest (10) members * [ ] Add expected impact to Reporter (20) members * [ ] Add expected impact to Developer (30) members * [ ] Add expected impact to Maintainer (40) members * [ ] Add expected impact to Owner (50) members --> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> - Unit and feature tests for new API endpoint are required for each parameter. Ensure each delete action does not delete trace files. - No End-to-End test required ### What does success look like, and how can we measure that? - A user is able to delete all erasable artifacts with one API request - The API endpoint is documented ### What is the type of buyer? <!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/ In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Is this a cross-stage feature? <!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features --> ### Links / references