Explore pipeline execution policy limits for customers using a single top-level group

Description

Today, we limit no more than 5 pipeline execution policies can be created within a single level of the GitLab structure. For instance, only 5 PEPs at the top level group, only 5 per sub-group. We also limit no more than 5 policies targeting a single project.

For many customers (such as self-managed customers), they leverage multiple top level groups. For GitLab.com and Dedicated they may also only have one top-level group. When moving from compliance pipelines, limits related to compliance frameworks that can be created with pipelines exceed that of the limits defined for PEP at the same level.

This issue will explore any suitable improvements to our limits to support customer use cases as well as ensuring performance/scalability.

Proposal

1. Provide a configuration option for Administrators to manage the limit of pipeline execution policies per security policy project.
2. Set a default limit of 5 pipeline execution policies, with an option to increase the limit to 20.
3. The limit of 5 pipeline execution policies per project would remain.
4. Self-managed customers would be able to modify this limit. Dedicated limits could be managed by GitLab administrators. GitLab.com would be continue to be limited to 5 and limits again managed by GitLab administrators.

Implementation Plan

1. backend Follow Add a new application setting to add 2 new application settings under the existing security_policies top-level key:

   • pipeline_execution_policies_per_configuration_limit
   • pipeline_execution_policies_per_pipeline_limit

2. backend Consume the application setting in place of current constants:

   2.1 Consume the pipeline_execution_policies_per_configuration_limit setting:

   diff --git a/ee/app/models/concerns/security/pipeline_execution_policy.rb b/ee/app/models/concerns/security/pipeline_execution_policy.rb
   index 8ac372dfa2b4..6d81016272b9 100644
   --- a/ee/app/models/concerns/security/pipeline_execution_policy.rb
   +++ b/ee/app/models/concerns/security/pipeline_execution_policy.rb
   @@ -2,15 +2,18 @@
   
    module Security
      module PipelineExecutionPolicy
   -    # This is the maximum number of PEPs in a policy config file
   -    POLICY_LIMIT = 5
   -
        def active_pipeline_execution_policies
   -      pipeline_execution_policy.select { |config| config[:enabled] }.first(POLICY_LIMIT)
   +      pipeline_execution_policy.select { |config| config[:enabled] }.first(policy_limit)
        end
   
        def pipeline_execution_policy
          policy_by_type(:pipeline_execution_policy)
        end
   +
   +    private
   +
   +    def policy_limit
   +      Gitlab::CurrentSettings.pipeline_execution_policies_per_configuration_limit
   +    end
      end
    end

   2.2 Consume the pipeline_execution_policies_per_pipeline_limit setting:

   diff --git a/ee/lib/gitlab/security/orchestration/project_pipeline_execution_policies.rb b/ee/lib/gitlab/security/orchestration/project_pipeline_execution_policies.rb
   index 3768dd4c2c37..0331865bf32d 100644
   --- a/ee/lib/gitlab/security/orchestration/project_pipeline_execution_policies.rb
   +++ b/ee/lib/gitlab/security/orchestration/project_pipeline_execution_policies.rb
   @@ -4,8 +4,6 @@ module Gitlab
      module Security
        module Orchestration
          class ProjectPipelineExecutionPolicies
   -        POLICY_LIMIT_PER_PIPELINE = 5
   -
            def initialize(project)
              @project = project
            end
   @@ -21,7 +19,7 @@ def initialize(project)
            #   Result: [policy5, policy4, policy3, policy2, policy1]
            def configs
              applicable_execution_policies_by_hierarchy
   -            .first(POLICY_LIMIT_PER_PIPELINE)
   +            .first(policy_limit)
                .reverse # reverse the order to apply the policy highest in the hierarchy as last
                .map do |(policy, policy_project_id, index)|
                  ::Security::PipelineExecutionPolicy::Config.new(
   @@ -51,6 +49,10 @@ def configs_ordered_by_hierarchy
                                                                                      .all.index_by(&:namespace_id)
              [nil, *@project.group&.self_and_ancestor_ids].filter_map { |id| configs[id] }.reverse
            end
   +
   +        def policy_limit
   +          Gitlab::CurrentSettings.pipeline_execution_policies_per_pipeline_limit
   +        end
          end
        end
      end

3. documentation Update pipeline_execution_policies.md to specifically note these settings in addition to regenerating Available settings