Improvements to NamespaceStorageApp app data

Sheldon Ledrequested to merge
led/426684-improve-storage_usage_app_data into master

What does this MR do and why?

This MR adds some changes in the storage_usage_app_data that will enable us to add some UI last details for https://gitlab.com/groups/gitlab-org/-/epics/11201

MR Description
!133516 (merged) (👈 you're here) Improvements to NamespaceStorageApp app data
!136463 (merged) Use Namespace Limits UI on pre-enforcement phase

Screenshots or screen recordings

These screenshots are mostly for guidance. We don't have UI changes per se but we'll have a behavior change in one of the scenarios below:

Group using the current `project limits` and under limit

133516_project-limits_under-limit

Group is now and over limit

133516_project-limits_over-limit

Group has purchased storage and is under limit again

133516_project-limits_under-limit-w-purchased-storage

Group is in preparation for the rollout. z-74 days scenario

133516_project-limits_no-limit

We start namespace limits pre-enforcement for the free Group. z-60 days scenario

133516_namespace-limits_pre-enforcement

We start namespace limits enforcement for the free Group Z-0 days of the rollout

In This MR we changed the value in the Included in Free subscription to always show the dashboard limit. If you're following the validation steps below you should see this difference:

In master In this MR's branch
133516_namespace-limits_enforcement-before 133516_namespace-limits_enforcement-after

How to set up and validate locally

Because this is a complex feature with a complex rollout, I'll be using the Rollout Timeline as the guide for how to test these changes.

The UI we're going to check is the Namespace Usage Quotas, accessible via: /<group-id>/-/usage_quotas#storage-quota-tab for your test group, and /-/profile/usage_quotas#storage-quota-tab for your test user namespace.

Setting up a Group for this test

Let's create or pick a group and a project under this group to go through the scenarios. Make note of the group and project IDs to use in rails console. To ease the manual test let's create some mock data using rails console:

  • Add some storage to your test project: Project.find(29).statistics.update(repository_size: 12.gigabytes)
  • Propagate this value to the group:
    • Sidekiq.redis(&:flushdb)
    • Namespaces::ScheduleAggregationWorker.perform_sync(89)
    • Namespaces::RootStatisticsWorker.perform_sync(89)

Testing current limits:

  • Make sure we have limits similar to prod: Gitlab::CurrentSettings.update!(repository_size_limit: 10.gigabytes)
  • The UI should now have a red alert like in the screenshot "Group is now and over limit" above

Testing purchased storage:

  • Add some purchased storage to the group: Group.find(89).update(additional_purchased_storage_size: 10*1024)
  • Note that the alert in the UI has disappeared, like in the screenshot "Group has purchased storage and is under limit again"

Testing z-74 days: Remove 10GiB per project limit

An important detail for this is that we'll need to set namespace limits for free namespaces like we did in prod: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8164. The difference here is that we'll use the UI:

  • Login in your GDK as root and go to this url: /admin/namespace_limits
  • In the "Dashboard Limit" section set the same value we have in prod: 5000
  • Click through the buttons to save the value.

Now Let's remove the project limits:

  • Login in your GDK as root and go to this url: /admin/application_settings/general
  • Expand the "Account and limit"
  • In the "Size limit per repository (MiB)" field, set it to 0
  • Check that your group/namespace UI has changed, like in the screenshots "Group is in preparation for the rollout"

Testing z-60 days: Pre-Enforcement Banners for free SaaS Namespaces

This part is under a Feature flag, so let's enable it in your GDK:

  • Feature.enable(:namespace_storage_limit_show_preenforcement_banner)

Now let's add a notification limit. Only namespaces over this limit will receive the pre-enforcement alert. The equation is actually total_storage > (notification_limit + purchased_storage). Let's mock this data:

  • Login in your GDK as root and go to this url: /admin/namespace_limits
  • In the "Notifications Limit" section set the value to 10240. This Means 10GiB
  • Click through the buttons to save the value.

Note that the UI hasn't changed. This is because the test group is under the notification_limit. If we replace the values in the equation it'd be 12GiB > (10GiB + 10GiB) so our test group needs to have more than 20GiB of used storage to start seeing the banner.

If you go to a different group and add 12GiB of storage, or remove the purchased storage of your group, you should see the banner. To match the screenshots of this MR let's not remove the purchased storage but rather increase total storage to 22GiB:

  • Add more storage to your test project: Project.find(29).statistics.update(repository_size: 22.gigabytes)
  • Propagate this value to the group:
    • Sidekiq.redis(&:flushdb)
    • Namespaces::ScheduleAggregationWorker.perform_sync(89)
    • Namespaces::RootStatisticsWorker.perform_sync(89)

Now you should see the yellow alert in the UI like in the screenshot: We start namespace limits pre-enforcement for the free Group. If you have a paid group check if UI matches the screenshot too.

Testing excluding a namespace from pre-enforcement and back

  • Login in your GDK as root and go to this url: /admin/namespace_limits
  • In the "Excluded Namespaces" section add your test group
  • Go back to the Usage Quotas UI and notice that the alert is gone
  • Back to /admin/namespace_limits delete your test group from the "Excluded Namespaces"
  • Go back to the Usage Quotas UI and notice that the alert is back

Testing Z-0 days: Storage Limits are Enforced for free SaaS Namespaces

This part is behind a feature flag and an application setting. Let's enable both in rails console:

  • Feature.enable(:namespace_storage_limit)
  • Gitlab::CurrentSettings.update!(enforce_namespace_storage_limit: true)

Now let's set an enforcement limit. Same as above, namespaces over this limit will be in a read-only state:

  • Login in your GDK as root and go to this url: /admin/namespace_limits
  • In the "Enforcement Limit" section set the value to 10240. This Means 10GiB
  • Click through the buttons to save the value.
  • Notice that the UI has changed as well as the alert, like in the screenshot "

Testing excluding a namespace from enforcement and back

  • Login in your GDK as root and go to this url: /admin/namespace_limits
  • In the "Excluded Namespaces" section add your test group
  • Go back to the Usage Quotas UI and notice that the alert is gone
    • Notice also that the cards changed, they're like in the "Group is in preparation for the rollout" screenshot
  • Back to /admin/namespace_limits delete your test group from the "Excluded Namespaces"
  • Go back to the Usage Quotas UI and notice that the alert is back

Done

These are all relevant scenarios for this rollout. For more information check our internal docs or the feature rollout issue

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sheldon Led

Merge request reports