Introduce storage counter at group level for package feature
Problem to solve
Package feature (maven, npm repositories) currently have no counter for used storage.
Intended users
All users who use Package feature.
Solution
Usage page
We will be reusing the existing Pipelines quota
page in group settings and renaming it to Usage quotas
. We will add a tab bar to allow users to switch between the existing view an a new Storage
view.
The title of the page will change to Usage quotas
and the short description to:
Usage of group resources across the projects in the [Name of group] group.
When the user selects the Storage
tab, we will show their total usage using the following format, alongside a progress bar:
[Used GB] / [Available GB]
This counter will have a ?
icon that will link to our docs explaining how storage is counted.
Below, we will display a table with all the projects in the group, with a Usage
column that will display how much storage each project is using.
Project details
Each project row can expand on click to show its detailed usage. Each sub-row will represent a 'concept' like packages, artifacts, container registry or LFS files. The usage for each concept will be displayed in storage units (MB, GB) as well as the percentage it represents in the project.
Removing the storage limit
Since we won't have storage limits in the first iteration, we cannot display the used storage as a Used / Available
. Therefore, we will present the total storage as Used GB
and forego the progress bar:
MVC
We will start by only counting the storage used by packages, so there will only be one concept for each project.
Note: We may have the backend capabilities to add more counters than just Packages in the first iteration and we will include them if possible, but listing Packages usage is our MVC.
This means that projects that don't have any packages will be shown as consuming 'O MB'. This will be displalyed using the same format:
Original proposal
Similar to pipeline minutes, introduce a counter at group level that will show the sum of all uploaded packages per project.
This interface will also need to contain other types of storage later, such as artifacts, container registry and lfs objects.
This proposal does not contain introducing storage limits at the group level which is part of https://gitlab.com/gitlab-org/gitlab-ce/issues/30421
What's needed here:
- Refactoring the "Pipelines quota" group page to likely simply read something like "Usage" or "Quotas".
- Considering how we display storage use at the group level. We can take an MVC approach as proposed in the description using the existing "progress bar" pattern, but we may want to consider a separate page (tab?) for group storage.
Permissions and Security
User who have access to the group settings (Maintainer and up).
Links / references
Development log
Status
-
frontend backend from "Pipeline quota" to "Usage quotas" gitlab!13139 (merged) -
backend MR for exposing statistics using Graphql API is in !28277 (merged) -
frontend MR for presenting the package stats and total stats is in gitlab!13294 (merged)
Decisions
- While working on !28277 (merged), it was discovered that performance of this query needs to be improved #62214 (closed)
- Because this needs to be done before we can expose the total storage usage for a group, this feature as it is defined now will possibly slip another milestone. To ensure that we ship parts that are actually usable we adapt the frontend to only expose individual storage data exposed by the backend (packages, lfs...) and not the total sum
-
We decided to use
N/A
as the total sum - while discussing about #62214 (closed) it was reported that the pattern we currently have with updating
project_statistics
doesn't scale well. Details in #62488 (moved) -
We will not include pagination on the first iteration.
- Frontend does not have a component that works with graphql #62788 (closed)
- backend pagination behaves erratically #62787 (closed)
The next iteration
- Performance
- Improve aggregation ~performance storing root-namespace storage statistics on database #62214 (closed) backend
- After performance improvement is done, expose the total storage counter sum gitlab#29262 (closed) backend
- Pagination
- Create a pagination component to handle graphql data #62788 (closed) frontend
- Fix GraphQL pagination gitlab#29267 backend
- Implement pagination gitlab#12058 (closed) frontend
- Implement "Usage Quotas > Storage" for personal projects gitlab#12060 (closed) backend frontend
- Write documentation for Usage Quotas #63022 (closed) ~Documentation