Backend: Recalculate remaining minutes on Additional Packs on a monthly basis

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem

With &5715 we now store any CI/CD minutes pack purchases as Ci::Minutes::AdditionalPack record. Each pack has an amount, purchase ID and an expiration date.

Currently these additional packs are populated on Gitlab SaaS but there is no logic that on a monthly basis recalculates the remaining minutes.

The existing logic that recalculates remaining minutes operates on namespaces.extra_shared_runners_minutes_limit column which is where the total number of additional minutes is stored today.

We have this data saved in 2 places:

  • namespaces.extra_shared_runners_minutes_limit is the legacy column where CustomersDot portal accumulates new purchases. On a monthly basis we reset the CI/CD minutes consumption and also recalculate what is the remaining amount of purchased minutes, if any used.
  • Ci::Minutes::AdditionalPack records is a side-by-side storage of CI/CD minutes purchased. On a monthly basis we DO NOT reset these yet. This is what's being asked in this issue.

Severity

This technical debt is set to severity2 because it represents a feature (AdditionalPacks) that is not yet been used. The customers, when purchasing additional minutes on the CustomersDot portal, are being told that the pack of additional minutes being purchased expires by X date, usually 1 year after purchase. Purchases of additional minutes increment the extra_shared_runners_minutes_limit (still used today) and create AdditionaPacks records, with expiration date, which are not used by the Rails application.

Soon (as purchased made with expiry date start to expire) this behavior will turn into a typebug (from a business perspective) because additional minutes that are supposed to expire, they won't.

Solution

To recalculate the amount in Ci::Minutes::AdditionalPacks we need to:

  1. sort non expired additional packs by expiration date and take the ones expiring sooner first.
  2. If any additional minutes have been used (total usage > monthly limit), reduce the amount from the first additional pack
  3. iterate with the following additional packs for the remaining amount.

Example:

Customer has used 10,000 minutes. Their monthly limit is 400. They have purchased the following packs:

  • Pack 1, amount 5,000 minutes, expiring January 2022 (expired)
  • Pack 2, amount 5,000 minutes, expiring March 2022
  • Pack 3, amount 5,000 minutes, expiring April 2022
  • Pack 4, amount 5,000 minutes, expiring June 2022

Given that the customer used 9,600 minutes more than their monthly limit (10,000 - 400), we need to reduce this amount from the additional packs:

  • Pack 1 remains as is because it's expired. The customer didn't use the minutes so they are lost.
  • Pack 2 becomes 0 (we still have 9600 - 5000 = 4600 minutes to reduce)
  • Pack 3 becomes 400 (5000 - 4600)
  • Pack 4 remains as is at 5,000.

Dependencies and rollout

Given that we already have additional packs on GitLab SaaS, the extra_shared_runners_minutes_limit does not match anymore with the actual amount in non expired additional packs. To make sure that legacy and new tracking of additional minutes remains consistent we need to follow the plan below:

  1. Disable temporarily the logic that creates additional packs from CustomersDot to GitLab SaaS.
  2. Remove existing packs on GitLab SaaS.
  3. grouphosted runners Implement logic behind a disabled feature flag that recalculates the remaining minutes in additional packs and updates extra_shared_runners_minutes_limit for the same amount.
  4. For each namespace with a non-zero extra_shared_runners_minute_limit, create an additional pack for that amount (without purchase XID, and with an arbitrary expiry date). This would ensure that extra_shared_runners_minute_limit and additional packs are in sync.
  5. grouphosted runners Enable the feature flag above since we now have additional packs.
  6. grouphosted runners Change Ci::Minutes::Limit to read purchased minutes as the sum of non-expired packs. This is important since a pack can expire at any time. This can be also done with a feature flag.
  7. Enable the logic that creates additional packs from CustomersDots to GitLab SaaS.
  8. Deprecate and remove extra_shared_runners_minute_limit.
  9. Remove any technical debt code introduced to fix cross-database modifications: #351847 (closed)

References

Edited by 🤖 GitLab Bot 🤖