Defer unenforceable policy evaluation with active pipeline
What does this MR do and why?
Fixes a race condition where UnenforceablePolicyRulesNotificationService prematurely unblocks fail-open approval rules.
!229073 (merged) added a pipeline_creating? guard in schedule_policy_synchronization to prevent enqueuing UnenforceablePolicyRulesNotificationWorker while the async CreatePipelineWorker hasn't finished. However, when the pipeline creation worker finishes before schedule_policy_synchronization runs, the Redis state is already cleared and as a result, the worker gets enqueued regardless and unblocks fail-open rules prematurely.
With this change, the service defers evaluation when a pipeline exists but is still running or the async pipeline creation worker got queued but hasn't created the pipeline record yet.
References
- Follow-up to !229073 (merged)
- Related to #578352 (closed)
Code references
The race condition:
Ci::CreatePipelineServiceclears Redis fromIN_PROGRESStoSUCCEEDEDbefore the worker writeshead_pipeline_idto databaseCreatePipelineWorkerwriteshead_pipeline_idto database afterCreatePipelineServicereturnsschedule_policy_synchronizationchecksdiff_head_pipeline(possibly stale in-memory association) andpipeline_creating?(falsesince Redis was cleared already)
The premature unblock:
pipelines_with_enforceable_reportsreturns[]becausecan_store_security_reports?requirescomplete_or_manual?can_store_security_reports?delegates tohas_security_reports?complete_or_manual_and_has_reports?requirescomplete_or_manual?which isfalsefor running pipelinesupdate_for_report_typecallsunblock_fail_open_ruleswhenpipelines_with_enforceable_reportsis empty
How to set up and validate locally
Same setup as !229073 (merged) (steps 1-5).
To trigger the race, add sleep 5 between prepare_for_mergeability and prepare_merge_request in AfterCreateService#execute. This ensures CreatePipelineWorker completes before schedule_policy_synchronization runs:
diff --git a/app/services/merge_requests/after_create_service.rb b/app/services/merge_requests/after_create_service.rb
index 83b17f03e690..fa27332a765c 100644
--- a/app/services/merge_requests/after_create_service.rb
+++ b/app/services/merge_requests/after_create_service.rb
@@ -6,6 +6,7 @@ def execute(merge_request)
merge_request.ensure_merge_request_diff
prepare_for_mergeability(merge_request)
+ sleep 5
prepare_merge_request(merge_request)
mark_merge_request_as_prepared(merge_request)Reproduce the bug on master
-
Open a new MR (add any file change)
-
Immediately check the violation status in Rails console:
mr = MergeRequest.last mr.scan_result_policy_violations.pluck(:id, :status) # => [[<id>, "warn"]]
Verify the fix
-
Repeat steps 1-2 on this branch
-
The violation should remain
running:mr = MergeRequest.last mr.scan_result_policy_violations.pluck(:id, :status) # => [[<id>, "running"]] -
The MR should NOT be mergeable while the pipeline is running
-
After the pipeline completes, the rule is evaluated normally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.