Drop running builds when CI minutes quota exceeded [RUN ALL RSPEC] [RUN AS-IF-FOSS]
What does this MR do?
related to https://gitlab.com/gitlab-org/gitlab/-/issues/20856
Feature flag issue: https://gitlab.com/gitlab-org/gitlab/-/issues/329197
How it works?
- Any time the Runner contacts Rails on
PUT /jobs/:id
orPATCH /jobs/:id/trace
we runCi::Minutes::TrackLiveConsumptionService
if:- we are on Gitlab.com and the project is on a Free plan namespace
- the project has
shared_runners_enabled: true
- the namespace has CI minutes limit defined
- This service saves on a Redis key the time a build consumption was tracked. Then, takes the remaining minutes from
Ci::Minutes::CachedQuota
, which are lazily instantiated in Redis with the value from the database and decrements them with the consumption since last tracking. - If the remaining minutes reaches quota
-1000
(arbitrary number to say that the customer used 1000 minutes more than their limit) then drop the build with failure reasonci_quota_exceeded
.
TODOs in follow-up MRs
- When minutes consumption is reset via the Admin controller, we also need to expire
minutes_used
inCachedQuota
- When
extra_shared_runners_minutes_limit
is changed via API (customer purchases additional minutes) we need to expireCachedQuota
- When
shared_runners_minutes_limit
is changed via API (plan change) we need to expireCachedQuota
- Compare actual consumption (in
UpdateBuildMinutesService
) with live consumption (from Redis entry) and log the difference in Prometheus so we can observe the accuracy of the this tracking. - Enable the Feature Flag
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because will be added when the feature flag is enabled.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Kamil Trzciński