Skip to content

Drop running builds when CI minutes quota exceeded [RUN ALL RSPEC] [RUN AS-IF-FOSS]

What does this MR do?

related to https://gitlab.com/gitlab-org/gitlab/-/issues/20856

Feature flag issue: https://gitlab.com/gitlab-org/gitlab/-/issues/329197

How it works?

  • Any time the Runner contacts Rails on PUT /jobs/:id or PATCH /jobs/:id/trace we run Ci::Minutes::TrackLiveConsumptionService if:
    • we are on Gitlab.com and the project is on a Free plan namespace
    • the project has shared_runners_enabled: true
    • the namespace has CI minutes limit defined
  • This service saves on a Redis key the time a build consumption was tracked. Then, takes the remaining minutes from Ci::Minutes::CachedQuota, which are lazily instantiated in Redis with the value from the database and decrements them with the consumption since last tracking.
  • If the remaining minutes reaches quota -1000 (arbitrary number to say that the customer used 1000 minutes more than their limit) then drop the build with failure reason ci_quota_exceeded.

TODOs in follow-up MRs

  1. When minutes consumption is reset via the Admin controller, we also need to expire minutes_used in CachedQuota
  2. When extra_shared_runners_minutes_limit is changed via API (customer purchases additional minutes) we need to expire CachedQuota
  3. When shared_runners_minutes_limit is changed via API (plan change) we need to expire CachedQuota
  4. Compare actual consumption (in UpdateBuildMinutesService) with live consumption (from Redis entry) and log the difference in Prometheus so we can observe the accuracy of the this tracking.
  5. Enable the Feature Flag

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Kamil Trzciński

Merge request reports