Allow remote CI configuration files for security policy compliance pipelines

We are introducing an experimental feature that allows to run custom CI configurations as part of a security policy. In the MVC version, the CI configuration has to be included in the policy.yml.

The goal of this feature is to replace the compliance pipelines feature. To archive this, it has to be able to load an external CI file. The policy should accept a ci_configuration_path filed that contains the path to the external CI file in a git repository.

We need to consider security implications: when the triggerer of the project pipeline does not have access to the project provided in ci_configuration_path we should not fetch that configuration and we should limit projects that you can fetch ci_configuration_path from to only these projects that share ancestor with project where policy is applied.

Feature Flag

remote_ci_configuration

Implementation plan

• Update security_orchestration_policy.json JSON schema to add definition for ci_configuration_path attribute:

diff --git a/ee/app/validators/json_schemas/security_orchestration_policy.json b/ee/app/validators/json_schemas/security_orchestration_policy.json
--- a/ee/app/validators/json_schemas/security_orchestration_policy.json	(revision 313934eb4dd666b38519dbd41f5f9577dd29a630)
+++ b/ee/app/validators/json_schemas/security_orchestration_policy.json	(date 1696011782582)
@@ -228,6 +228,9 @@
                 },
                 "ci_configuration": {
                   "type": "string"
+                },
+                "ci_configuration_path": {
+                  "type": "string"
                 }
               },
               "allOf": [
@@ -240,8 +243,17 @@
                     }
                   },
                   "then": {
-                    "required": [
-                      "ci_configuration"
+                    "anyOf": [
+                      {
+                        "required": [
+                          "ci_configuration"
+                        ]
+                      },
+                      {
+                        "required": [
+                          "ci_configuration_path",
+                        ]
+                      }
                     ],
                     "maxProperties": 2
                   }           

• Update Security::SecurityOrchestrationPolicies::CiConfigurationService to receive the user triggering the pipeline, and the project
• Update Security::SecurityOrchestrationPolicies::CiConfigurationService to load an external CI file

===================================================================
diff --git a/ee/app/services/security/security_orchestration_policies/ci_configuration_service.rb b/ee/app/services/security/security_orchestration_policies/ci_configuration_service.rb
--- a/ee/app/services/security/security_orchestration_policies/ci_configuration_service.rb	(revision 313934eb4dd666b38519dbd41f5f9577dd29a630)
+++ b/ee/app/services/security/security_orchestration_policies/ci_configuration_service.rb	(date 1696019481591)
@@ -26,8 +26,16 @@
       private
 
       def custom_pipeline_configuration(action, index)
-        Gitlab::Ci::Config.new(action[:ci_configuration]).to_hash
+        if action[:ci_configuration]
+          Gitlab::Ci::Config.new(action[:ci_configuration]).to_hash
+        elsif action[:ci_configuration_path] && Feature.enabled?(:remote_ci_configuration, project)
+          Gitlab::Ci::Config.new(external_config_content).to_hash if can_load_external_config?
+        end
       rescue Gitlab::Ci::Config::ConfigError => e
+        custom_configuration_error(e, index)
+      end
+
+      def custom_configuration_error(e, index)
         {
           generate_job_name_with_index('security_policy_ci', index) => {
             'script' => "echo \"Error parsing security policy CI configuration: #{e.message}\" && false",
@@ -36,6 +44,10 @@
         }
       end
 
+      def can_load_external_config?
+        can?(current_user, :read_project, config_project) && config_project.root_ancestor == project.root_ancestor
+      end
+
       def scan_template(scan_type)
         template = ::TemplateFinder.build(:gitlab_ci_ymls, nil, name: SCAN_TEMPLATES[scan_type]).execute
         Gitlab::Ci::Config.new(template.content).to_hash

To load the external config, maybe we can do something similar to lib/gitlab/ci/project_config/external_project.rb but avoiding adding the CI yaml using include: external_config as suggested here. This class also contains some logic to extract the project from the config file path. Something that we will need for the validations.