Prevent Git Push When Namespace Storage Limit Exceeded
What does this MR do and why?
Prevent git push when namespace storage limit is exceeded.
Bypass repository storage limits if namespace storage limits are enforced.
Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/354363
How to set up and validate locally
It may be beneficial to test using git push over both HTTP and SSH.
Validate git push rejected when namespace storage limits are exceeded
- Make sure the
automatic_purchased_storage_allocationandenforce_namespace_storage_limitapplication settings are both enabled (true).
gitlabhq_development=# SELECT automatic_purchased_storage_allocation, enforce_namespace_storage_limit FROM application_settings;
automatic_purchased_storage_allocation | enforce_namespace_storage_limit
----------------------------------------+---------------------------------
t | t
(1 row)
gitlabhq_development=# - Enable the
:namespace_storage_limit,:enforce_storage_limit_for_paid,:enforce_storage_limit_for_free, and:namespace_storage_limit_bypass_date_checkfeature flags in a rails console.
[1] pry(main)> Feature.enable(:namespace_storage_limit)
[...]
[2] pry(main)> Feature.enable(:enforce_storage_limit_for_paid)
[...]
[3] pry(main)> Feature.enable(:enforce_storage_limit_for_free)
[...]
[4] pry(main)> Feature.enable(:namespace_storage_limit_bypass_date_check)
[...]
[5] pry(main)> - Apply the following patch to your local gitlab instance. This patch disables some caching around storage size and storage limits.
diff --git a/ee/app/models/ee/namespace/root_storage_size.rb b/ee/app/models/ee/namespace/root_storage_size.rb
index fd612db00ec..b37918bdfbc 100644
--- a/ee/app/models/ee/namespace/root_storage_size.rb
+++ b/ee/app/models/ee/namespace/root_storage_size.rb
@@ -26,16 +26,16 @@ def usage_ratio
end
def current_size
- @current_size ||= Rails.cache.fetch(['namespaces', root_namespace.id, CURRENT_SIZE_CACHE_KEY], expires_in: EXPIRATION_TIME) do
+ # @current_size ||= Rails.cache.fetch(['namespaces', root_namespace.id, CURRENT_SIZE_CACHE_KEY], expires_in: EXPIRATION_TIME) do
root_namespace.root_storage_statistics&.storage_size
- end
+ # end
end
def limit
- @limit ||= Rails.cache.fetch(['namespaces', root_namespace.id, LIMIT_CACHE_KEY], expires_in: EXPIRATION_TIME) do
+ # @limit ||= Rails.cache.fetch(['namespaces', root_namespace.id, LIMIT_CACHE_KEY], expires_in: EXPIRATION_TIME) do
root_namespace.actual_limits.storage_size_limit.megabytes +
root_namespace.additional_purchased_storage_size.megabytes
- end
+ # end
end
def remaining_storage_percentage-
Restart your GDK simulating saas mode with
$ GITLAB_SIMULATE_SAAS=1 gdk start. -
Pick a local project to test with. Clone the project. Create a branch and add a new file. Push the new branch.
$ git checkout -b my-test-branch
Switched to a new branch 'my-test-branch'
$ echo "Add some text here" > new-file.txt
$ git add .
$ git commit -m 'Make a new file'
[my-test-branch 6d67b0d] Make a new file
1 file changed, 1 insertion(+)
create mode 100644 new-file.txt
$ git push -u local-gitlab my-test-branch
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 12 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 296 bytes | 296.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for my-test-branch, visit:
remote: http://local-gitlab:3000/storage-test-group/storage-test-project/-/merge_requests/new?merge_request%5Bsource_branch%5D=my-test-branch
remote:
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
* [new branch] my-test-branch -> my-test-branch
branch 'my-test-branch' set up to track 'local-gitlab/my-test-branch'.
$ - Navigate to your project's group page. In the example here, my project is "Storage Test Project" under the "Storage Test Group" group so I navigate to the "Storage Test Group" page, pictured below.
Navigate to Settings > Usage Quotas. Click the Storage tab. You can see how much total namespace storage you currently have here.
This can give you a good idea of what number to use in the SQL UPDATE statement below in step 8. Pick something below the total namespace storage.
- Find the
plan_limit.idof your localplan_limitfor thefreeplan. (The project in these examples is in a group on the free plan. You may need to use a different plan if your project is on an Ultimate or Premium plan.)
gitlabhq_development=# SELECT p.id, p.name, p.title, l.id, l.storage_size_limit FROM plans AS p JOIN plan_limits AS l ON p.id = l.plan_id;
id | name | title | id | storage_size_limit
----+----------------+---------------------------+----+--------------------
1 | default | Default | 1 | 0
2 | bronze | Bronze | 6 | 0
3 | silver | Silver | 7 | 0
4 | gold | Gold | 8 | 0
5 | free | | 5 | 3
6 | premium | Premium (Formerly Silver) | 23 | 0
7 | ultimate | Ultimate (Formerly Gold) | 26 | 0
8 | ultimate_trial | Ultimate Trial | 3 | 0
9 | premium_trial | Premium Trial | 4 | 0
10 | opensource | Open Source Program | 28 | 0
(10 rows)
gitlabhq_development=# - Update your local plan limits for the
freeplan. Your id may be different than the one seen below.
gitlabhq_development=# UPDATE plan_limits SET storage_size_limit = 3 WHERE id = 5;
UPDATE 1
gitlabhq_development=# - Log in as an administrator. Navigate to the Admin section. Navigate to Settings > General. Expand the Account and limit section. Make sure your Size limit per repository (MB) is set to either 0 or a very high number.
This is just to make sure that you're under the repository size limits to demonstrate that these don't matter anymore with the changes in this MR.
- Try to make a change to the file you created above.
git pushthe change. The push is rejected.
$ echo 'change file' > new-file.txt
$ git add .
$ git commit -m 'Update file'
[my-test-branch ad9d656] Update file
1 file changed, 1 insertion(+), 1 deletion(-)
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 267 bytes | 267.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: GitLab: Your push to this repository has been rejected because the namespace storage limit of 3 MB has been reached. Reduce your namespace storage or purchase additional storage.
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
! [remote rejected] my-test-branch -> my-test-branch (pre-receive hook declined)
error: failed to push some refs to 'http://local-gitlab:3000/storage-test-group/storage-test-project.git'
$ - Update the namespace storage limits. Set the limit to 1 MB above the total namespace storage.
gitlabhq_development=# UPDATE plan_limits SET storage_size_limit = 12 WHERE id = 5;
UPDATE 1
gitlabhq_development=# SELECT p.id, p.name, p.title, l.id, l.storage_size_limit FROM plans AS p JOIN plan_limits AS l ON p.id = l.plan_id;
id | name | title | id | storage_size_limit
----+----------------+---------------------------+----+--------------------
1 | default | Default | 1 | 0
2 | bronze | Bronze | 6 | 0
3 | silver | Silver | 7 | 0
4 | gold | Gold | 8 | 0
5 | free | | 5 | 12
6 | premium | Premium (Formerly Silver) | 23 | 0
7 | ultimate | Ultimate (Formerly Gold) | 26 | 0
8 | ultimate_trial | Ultimate Trial | 3 | 0
9 | premium_trial | Premium Trial | 4 | 0
10 | opensource | Open Source Program | 28 | 0
(10 rows)
gitlabhq_development=# - Try to
git pushthe file again. The push is accepted.
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 267 bytes | 267.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote:
remote: ========================================================================
remote:
remote: ##### WARNING ##### You have reached 93% of Storage Test Group's
remote: storage capacity (11 MB of 12 MB) If you reach 100% storage
remote: capacity, you will not be able to: push to your repository, create
remote: pipelines, create issues or add comments. To reduce storage
remote: capacity, delete unused repositories, artifacts, wikis, issues, and
remote: pipelines.
remote:
remote: ========================================================================
remote:
remote:
remote: To create a merge request for my-test-branch, visit:
remote: http://local-gitlab:3000/storage-test-group/storage-test-project/-/merge_requests/new?merge_request%5Bsource_branch%5D=my-test-branch
remote:
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
2b5f404..825784f my-test-branch -> my-test-branch
$ Note: The warning message seen above is outside the scope of this issue. It has its own issue: https://gitlab.com/gitlab-org/gitlab/-/issues/360326
- Create a 2 MB file of random data. Commit the file and try to
git pushto the remote. The push is rejected. Notice the error message says that this is because the push size would cause the namespace storage limits to be exceeded.
$ head -c 2048000 /dev/random > my-2mb-file
$ git add my-2mb-file
$ git commit -m 'Add a 2 MB file'
[my-test-branch a7b1bcf] Add a 2 MB file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 my-2mb-file
$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.95 MiB | 1.85 MiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: GitLab: Your push to this repository has been rejected because it would exceed the namespace storage limit of 12 MB. Reduce your namespace storage or purchase additional storage.
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
! [remote rejected] my-test-branch -> my-test-branch (pre-receive hook declined)
error: failed to push some refs to 'http://local-gitlab:3000/storage-test-group/storage-test-project.git'
$
Validate project repository limits are ignored for git push
- Update the namespace storage limits to a value higher than the total namespace storage.
gitlabhq_development=# UPDATE plan_limits SET storage_size_limit = 50 WHERE id = 5;
UPDATE 1
gitlabhq_development=# SELECT p.id, p.name, p.title, l.id, l.storage_size_limit FROM plans AS p JOIN plan_limits AS l ON p.id = l.plan_id;
id | name | title | id | storage_size_limit
----+----------------+---------------------------+----+--------------------
1 | default | Default | 1 | 0
2 | bronze | Bronze | 6 | 0
3 | silver | Silver | 7 | 0
4 | gold | Gold | 8 | 0
5 | free | | 5 | 50
6 | premium | Premium (Formerly Silver) | 23 | 0
7 | ultimate | Ultimate (Formerly Gold) | 26 | 0
8 | ultimate_trial | Ultimate Trial | 3 | 0
9 | premium_trial | Premium Trial | 4 | 0
10 | opensource | Open Source Program | 28 | 0
(10 rows)
gitlabhq_development=# - Set the repository size limit to
1.
- Create a new branch. Create and commit a new file on the branch.
git pushthe branch to the remote. The push is accepted.
$ git checkout master
$ git checkout -b new-test-branch
Switched to a new branch 'new-test-branch'
$ echo 'another file' > another-file.txt
$ git add .
$ git commit -m 'Add another file'
[new-test-branch c9af92a] Add another file
1 file changed, 1 insertion(+)
create mode 100644 another-file.txt
$ git push -u local-gitlab new-test-branch
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 12 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 365 bytes | 365.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for new-test-branch, visit:
remote: http://local-gitlab:3000/storage-test-group/storage-test-project/-/merge_requests/new?merge_request%5Bsource_branch%5D=new-test-branch
remote:
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
* [new branch] new-test-branch -> new-test-branch
branch 'new-test-branch' set up to track 'local-gitlab/new-test-branch'.
$ Validate branches can be deleted when over namespace storage limits
- Update the namespace storage limits. Set the limit to a number below your currently used storage.
gitlabhq_development=# UPDATE plan_limits SET storage_size_limit = 3 WHERE id = 5;
UPDATE 1
gitlabhq_development=# - Set your project repository size limits to either 0 or a very high number.
- Try to
git pushto verify that the push is blocked by the namespace storage limits again.
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 307 bytes | 307.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: GitLab: Your push to this repository has been rejected because the namespace storage limit of 3 MB has been reached. Reduce your namespace storage or purchase additional storage.
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
! [remote rejected] somebranch -> somebranch (pre-receive hook declined)
error: failed to push some refs to 'http://local-gitlab:3000/storage-test-group/storage-test-project.git'
$ - Try to delete a branch with
git push -d. The branch is deleted.
$ git push -d local-gitlab my-test-branch
remote:
remote: ========================================================================
remote:
remote: ##### ERROR ##### You have reached 375% of Storage Test Group's
remote: storage capacity (11 MB of 3 MB) Storage Test Group is now
remote: read-only. You cannot: push to your repository, create pipelines,
remote: create issues or add comments. To reduce storage capacity, delete
remote: unused repositories, artifacts, wikis, issues, and pipelines.
remote:
remote: ========================================================================
remote:
To http://local-gitlab:3000/storage-test-group/storage-test-project.git
- [deleted] my-test-branch
$ The ERROR message seen above is outside the scope of this issue and will be handled in https://gitlab.com/gitlab-org/gitlab/-/issues/360326.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Jason Goodman