Adding CI minutes consumption is not idempotent
Problem
When a job finishes we calculate the CI minutes consumption and add it to the total consumption for the relative Namespace and Project.
This logic, however, runs synchronously as part of BuildFinishedWorker
. Since this worker triggers a lot of logic, both inline and async it can fail typically because of the inline logic causing the worker to rerun and re-add CI minutes to the namespace consumption.
Solution
Ensure that all async job executions are done after the sync operations from BuildFinishedWorker
diff --git a/ee/app/workers/ee/ci/build_finished_worker.rb b/ee/app/workers/ee/ci/build_finished_worker.rb
index 05679955b4a..33e93ffd82f 100644
--- a/ee/app/workers/ee/ci/build_finished_worker.rb
+++ b/ee/app/workers/ee/ci/build_finished_worker.rb
@@ -4,6 +4,8 @@ module EE
module Ci
module BuildFinishedWorker
def process_build(build)
+ super
+
unless ::Feature.enabled?(:cancel_pipelines_prior_to_destroy, default_enabled: :yaml)
::Ci::Minutes::UpdateBuildMinutesService.new(build.project, nil).execute(build)
end
@@ -15,8 +17,6 @@ def process_build(build)
if ::Gitlab.com? && build.has_security_reports?
::Security::TrackSecureScansWorker.perform_async(build.id)
end
-
- super
end
end
end
Ensure we track the operation as done in Redis so that other attempts for the same build ID are not performed.
-
Introduce a new param build_id
that triggers the idempotent logic - !68861 (merged) -
Pass build_id
to the worker - !69994 (merged) -
Remove the default value nil
forbuild_id
param as per https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#multi-step-deployment -
Rollout feature flag idempotent_ci_minutes_consumption
Edited by Fabio Pitino