Count duration of shared runners usage separately from CI minutes for viewing of consumption
Problem to solve
There has been a lot of confusion by open source projects and contributors about the latest announcement to cut CI minutes for Free users. Public projects' CI minutes are not being counted, and this is being called a
bug: #243722 (closed)
To my understanding, we decided to call it a bug for now in order to buy time. We want to consider the way that the announcement affects open source projects and contributors. See: https://gitlab.com/gitlab-com/packaging-and-pricing/pricing-handbook/-/issues/204
[effectively] unlimited CI mins for public projects be classified as a
bug makes it seem like the bug can be fixed at any time without sufficient warning to users.
In summary, these are the challenges associated with calling this a
- No visibility on CI minute usage by public projects. CI minutes continue to not be counted for public projects and this means there is no visibility on how many minutes are being used. Projects can't make adjustments easily without this visibility, and we aren't able to inform any policies with actual usage data.
- Open source contributions are in jeopardy. Some open source projects require that contributors test out their contributions using CI/CD before submitting (see comment from user here). Others like the Samba project, a large and well-known open source project/community, would not be able to have open source contributions made to their project if the new limits apply to them (see comment from user here).
The intended users would be, primarily, open source projects hosted on GitLab.com and their contributors.
User experience goal
We would want public project account holders to be able to view their CI minute usage, but not have it count against their limit. It would effectively be "for display only" as @fabiopitino mentions here: #243722 (comment 415924742)
Today the monthly charts will display the CI minutes consumption (in terms of credits/points) and not in terms of duration, because they are multiplied by the cost factor at the time the jobs completed.
Perhaps we could add a new column in
ci_project_monthly_usage to accumulate the "duration in minutes" for all projects/namespaces. This value would be reset automatically on a monthly basis and it won't be used to enforce CI minutes consumption since we already have a column for that that takes in consideration the cost factor.
Once we track the duration of all shared runners usage per namespace/project we can make better predictions on limits. E.g. A namespace using on average 1000 minutes would be billed as:
- up to 1000 CI minutes if it uses only private projects
- as low as 8 CI minutes if it uses only public projects (current cost factor of 0.008)
- about 166 CI minutes if it uses only MacOS runners (assuming a cost factor of 6.0)
We need to:
- add new column
shared_runners_minutes_durationcolumn to differ from #338830)
- add new column
shared_runners_minutes_durationcolumn to differ from #338830)
- track duration side by side to CI minutes consumption
Draft of a solution
diff --git a/ee/app/models/ci/minutes/namespace_monthly_usage.rb b/ee/app/models/ci/minutes/namespace_monthly_usage.rb index 064355a4f03..4e55edc38ca 100644 --- a/ee/app/models/ci/minutes/namespace_monthly_usage.rb +++ b/ee/app/models/ci/minutes/namespace_monthly_usage.rb @@ -27,13 +27,13 @@ def self.find_or_create_current(namespace_id:) current_month.safe_find_or_create_by(namespace_id: namespace_id) end - def self.increase_usage(usage, amount) + def self.increase_usage(usage, amount, duration = 0) return unless amount > 0 # The use of `update_counters` ensures we do a SQL update rather than # incrementing the counter for the object in memory and then save it. # This is better for concurrent updates. - update_counters(usage, amount_used: amount) + update_counters(usage, amount_used: amount, duration: duration) end end end diff --git a/ee/app/models/ci/minutes/project_monthly_usage.rb b/ee/app/models/ci/minutes/project_monthly_usage.rb index ad134c36aa4..39d87e11668 100644 --- a/ee/app/models/ci/minutes/project_monthly_usage.rb +++ b/ee/app/models/ci/minutes/project_monthly_usage.rb @@ -30,13 +30,13 @@ def self.find_or_create_current(project_id:) current_month.safe_find_or_create_by(project_id: project_id) end - def self.increase_usage(usage, amount) + def self.increase_usage(usage, amount, duration = 0) return unless amount > 0 # The use of `update_counters` ensures we do a SQL update rather than # incrementing the counter for the object in memory and then save it. # This is better for concurrent updates. - update_counters(usage, amount_used: amount) + update_counters(usage, amount_used: amount, duration: duration) end end end diff --git a/ee/app/services/ci/minutes/update_build_minutes_service.rb b/ee/app/services/ci/minutes/update_build_minutes_service.rb index 9b5463f2f94..b5e123f87f1 100644 --- a/ee/app/services/ci/minutes/update_build_minutes_service.rb +++ b/ee/app/services/ci/minutes/update_build_minutes_service.rb @@ -14,17 +14,17 @@ def execute(build) - return unless build.cost_factor_enabled? return unless build.complete? return unless build.duration&.positive? + return unless build&.runner&.instance_type? consumption = ::Gitlab::Ci::Minutes::BuildConsumption.new(build, build.duration).amount - return unless consumption > 0 - update_minutes(consumption) + update_minutes(consumption, build.duration) compare_with_live_consumption(build, consumption) end private - def update_minutes(consumption) + def update_minutes(consumption, duration) if ::Feature.enabled?(:cancel_pipelines_prior_to_destroy, project, default_enabled: :yaml) - ::Ci::Minutes::UpdateProjectAndNamespaceUsageWorker.perform_async(consumption, project.id, namespace.id) + ::Ci::Minutes::UpdateProjectAndNamespaceUsageWorker.perform_async(consumption, project.id, namespace.id, duration) else - ::Ci::Minutes::UpdateProjectAndNamespaceUsageService.new(project.id, namespace.id).execute(consumption) + ::Ci::Minutes::UpdateProjectAndNamespaceUsageService.new(project.id, namespace.id).execute(consumption, duration) end end diff --git a/ee/app/services/ci/minutes/update_project_and_namespace_usage_service.rb b/ee/app/services/ci/minutes/update_project_and_namespace_usage_service.rb index 831eb224b41..8b362170760 100644 --- a/ee/app/services/ci/minutes/update_project_and_namespace_usage_service.rb +++ b/ee/app/services/ci/minutes/update_project_and_namespace_usage_service.rb @@ -13,10 +13,10 @@ def initialize(project_id, namespace_id) end # Updates the project and namespace usage based on the passed consumption amount - def execute(consumption) + def execute(consumption, duration = nil) legacy_track_usage_of_monthly_minutes(consumption) ApplicationRecord.transaction do - track_usage_of_monthly_minutes(consumption) + track_usage_of_monthly_minutes(consumption, duration) send_minutes_email_notification end @@ -37,12 +37,12 @@ def legacy_track_usage_of_monthly_minutes(consumption) update_legacy_namespace_minutes(consumption_in_seconds) end - def track_usage_of_monthly_minutes(consumption) + def track_usage_of_monthly_minutes(consumption, duration) # TODO(issue 335885): Remove @project return unless Feature.enabled?(:ci_minutes_monthly_tracking, @project, default_enabled: :yaml) - ::Ci::Minutes::NamespaceMonthlyUsage.increase_usage(namespace_usage, consumption) if namespace_usage - ::Ci::Minutes::ProjectMonthlyUsage.increase_usage(project_usage, consumption) if project_usage + ::Ci::Minutes::NamespaceMonthlyUsage.increase_usage(namespace_usage, consumption, duration) if namespace_usage + ::Ci::Minutes::ProjectMonthlyUsage.increase_usage(project_usage, consumption, duration) if project_usage end def update_legacy_project_minutes(consumption_in_seconds) diff --git a/ee/app/workers/ci/minutes/update_project_and_namespace_usage_worker.rb b/ee/app/workers/ci/minutes/update_project_and_namespace_usage_worker.rb index fd97240903f..4f0f8e7a517 100644 --- a/ee/app/workers/ci/minutes/update_project_and_namespace_usage_worker.rb +++ b/ee/app/workers/ci/minutes/update_project_and_namespace_usage_worker.rb @@ -9,8 +9,8 @@ class UpdateProjectAndNamespaceUsageWorker # rubocop:disable Scalability/Idempot urgency :low data_consistency :always # primarily performs writes - def perform(consumption, project_id, namespace_id) - ::Ci::Minutes::UpdateProjectAndNamespaceUsageService.new(project_id, namespace_id).execute(consumption) + def perform(consumption, project_id, namespace_id, duration = nil) + ::Ci::Minutes::UpdateProjectAndNamespaceUsageService.new(project_id, namespace_id).execute(consumption, duration) end end end
As written in issue: #243722 (comment 415905370)...
Here are some proposed next steps for how to deal with this bug:
Revisit our policy about public projects and CI/CD minute accumulation in relation to open source contribution. We should make sure that we do not penalize people for contributing to open source projects or hosting their open source projects on GitLab. Instead, we should encourage this activity. We have heard from focus groups and users on our forum that sometimes open source projects require contributors to test out their contributions (run CI), which means some Free users will need to use up their own CI mins.
Fix this bug. It's useful for everyone (including us), to understand how many CI minutes are being used by public projects. People can't make adjustments easily if they don't have visibility into how many CI minutes they are using. When we fix this bug, we should not enforce CI minute limits on public projects for at least 3 months, if at all, in order to allow people and projects to adjust. (Deciding on the amount of time we allow for transition, if relevant, should be part of step 1)
Enforce limits. We should enforce our policy only once step 1 and 2 are complete and there has been sufficient adjustment time permitted.
Summary of analysis to date (revised 2020-10-30):
- The data sources in Sisense for the cost impact estimates are:
- How to read the Sisense report:
- CI MINUTES: The total minutes used per plan per month.
- OVERAGE MINUTES: The total overage minutes incurred per plan per month.
- Average monthly CI minutes used by public projects in FY21 to date = 12.8m
- Average monthly overage minutes used by public projects in FY21 = 11m (so the overage rate is at 85%)
- Public projects in the free plan account for 82% of the overages.