Storage limits for the Dependency Proxy
Problem to solve
The Dependency Proxy allows users to reduce risk and reliance on external dependencies by caching frequently used packages for fast, reliable access. The current implementation allows users to download dependencies from DockerHub and store them either locally or using their own object storage. As adoption grows, we expect the amount of storage being used to grow rapidly. However, we do not provide administrators the ability to monitor usage, set instance or group level size limits.
Not providing these mechanisms for control, creates risk for our customers. First, they may be surprised by increased storage costs and lack the controls to better manage those costs. Second, while there are no size limitations, we are limited by web servers and network bandwidth, which could lead to a DDoS attack of the storage for their GitLab instance.
Intended users
Proposal
Provide administrators the ability to set a storage quota per group and a total per instance for the dependency proxy. We already provide this functionality for repositories at /admin/application_settings
but should extend it to cover the dependency proxy as well.
An admin should be able to find these settings in a group's settings or instance settings. We will add two fields:
-
Dependency Proxy size limit (instance)
: Default to 0 (no limit) -
Dependency Proxy size limit per group
: Default to 10 GB
Further details
Alerts
- If a user attempts to go over the allotted amount of storage for a particular group or instance, we will alert that user that upload failed due to storage restrictions and prompt them to contact their administrator.
- If a group or instance level limit has been met, we will alert the administrator that their limit has been met and prompt them to allot more storage or delete older, unused items from the dependency proxy.
Usage quotas
https://gitlab.com/gitlab-org/gitlab-ce/issues/59232 introduces a new user interface that will display storage usage of features like the GitLab Container Registry or Dependency Proxy. We will add the dependency proxy to this section so users can see how much storage is being used at the instance/group/project level and how that compares to what they've allotted.
Permissions and Security
This functionality will only be available for instance Owners.
Documentation
- Admin: account and limit settings must be updated to include dependency proxy details
- Dependency Proxy Configuration Docs will be updated
Testing
- Ensure that notifications are sent and received
- Ensure that exceeding limitations does not cause any unforeseen problems
What does success look like, and how can we measure that?
Metrics
- Track group and instance level limits: This will allow us to set better defaults in the future
- number of events where limit was exceeded
- number of events where limits were adjusted by an admin
- number of views of usage quotas page
Success looks like we give our users the ability to control and manage their storage costs and prevent any potential DDoS attacks.
Future Iterations
- Ability to delete / purge items from the proxy
- Ability to define and automatically act on policies for garbage collection, expiration and purging
- Use
Rack::Attack
or other way to limit queries on dependency_proxy endpoint
What is the type of buyer?
For now, the dependency proxy is focused on premium and ultimate customers.
In order to expand to additional buyer types we need to improve performance and support Unicorn. This issue details those efforts: https://gitlab.com/gitlab-org/gitlab-ee/issues/11548
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.