Validate, accompany with enabling `policy_mergability_check` feature flag and address identified problems

Background

The policy_mergability_check feature flag has been tested, and some issues were encountered during its rollout. This issue aims to address the identified problems and prepare for successfully enabling the feature flag.

Objectives

  1. Add logging to improve visibility into potential blocking cases
  2. Address identified edge cases
  3. Plan and execute a controlled rollout of the feature flag

Tasks

1. Improve Logging

  • Add logging at points where policy evaluation is expected to be finished
  • Log information when an MR gets merged with a finished or no pipeline, and there are violations stuck in running state
  • Include policy options (fail-open vs. fail-closed), vulnerability states, and whether a pipeline had security artifacts in the logs

2. Address Edge Cases

  • Handle manual pipelines correctly (ref: Consider manual pipeline for unenforceable poli... (!172570 - merged))
  • Ensure fail-open policies work as expected when no security jobs run
  • Verify behavior with skip ci and when "pipeline must succeed" is false
  • Test and fix any issues with both branch & merge request pipelines in the MR, where MR pipelines don't produce scans

3. Rollout Plan

  • Monitor logs for a defined period to identify any remaining edge cases
  • Create a phased rollout plan (e.g., start with a small percentage of projects or specific groups)
  • Define criteria for progressing through rollout phases
  • Prepare communication for users about the feature and potential impact

4. Testing

  • Create comprehensive test scenarios covering various pipeline configurations and policy settings
  • Perform thorough testing in a staging environment before each rollout phase

5. Monitoring and Quick Response

  • Set up monitoring for any blocking cases or unexpected behavior
  • Prepare a rollback plan in case of significant issues
  • Assign team members to monitor and respond quickly to any reported problems during the rollout

Notes

  • The feature flag should never cause an MR to be permanently blocked. If such a case is identified, it should be treated as a high-priority bug.
  • Consider the impact on users with fail-open policies and ensure they are not negatively affected by the changes.
Edited by Martin Cavoj