Backfill security policies table from YAML

What does this MR do and why?

The security policies are stored as YAML files in the security policy project. This approach has a lot of advantages (like version control for policies using git, auditable etc), it faces some performance drawbacks. To improve this we are storing security policy data in the security_policies table. This MR adds a migration to backfill data reading from .gitlab/security-policies/policy.yml files.

1. It iterates through each rows of security_orchestration_policy_configurations
2. read the policy YAML from gitaly
3. validates the policy if it matches the schema
4. for each policy type and policy, persist the policy to security_policies table
5. It iterate through the rules of policy with types approval_policy, scan_execution_policy, vulnerability_management_policy and persist the rules to approval_policy_rules, scan_execution_policy_rules & vulnerability_management_policy_rules correspondingly
6. If the security_orchestration_policy_configuration is mapped to a namepsace, get all projects in the namespace, and for each project check the policy_scope similar to Security::SecurityOrchestrationPolicies::PolicyScopeChecker and create entries in security_policy_project_links & approval_policy_rule_project_links

Database

We usually isolate background migrations from the rest of the application. In this MR there are two exceptions:

1. Use ee/app/validators/json_schemas/security_orchestration_policy.json to validate the security policy schema. We take care not to introduce breaking changes outside of major milestones so I think it is okay to validate against this schema.
2. ::Gitlab::Git classes. We have to access git to load yaml files from repositories. Those classes are a core part of GitLab. Copying this to the background migration will make it significantly larger. Also, we will have to rely on gitaly eventually, so there is still an external dependency.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to setup locally?

• Create a group (read-model-migration) and create multiple sub-groups and projects
• Disable security_policies_sync_group feature flag (Feature.disable(:security_policies_sync_group))
• Create a project (read-model-migration/read-model-migration-security-policy-project) within the group and create .gitlab/security-policies/policy.yml file with:

---
approval_policy:
- name: MRAP - Security Scan
  description: ''
  enabled: true
  policy_scope:
    projects:
      excluding: []
  rules:
  - type: scan_finding
    scanners: []
    vulnerabilities_allowed: 0
    severity_levels: []
    vulnerability_states: []
    branch_type: protected
  actions:
  - type: require_approval
    approvals_required: 1
    role_approvers:
    - developer
  - type: send_bot_message
    enabled: true
  approval_settings:
    block_branch_modification: true
    block_group_branch_modification: true
    prevent_pushing_and_force_pushing: true
    prevent_approval_by_author: true
    prevent_approval_by_commit_author: true
    remove_approvals_with_new_commit: true
    require_password_to_approve: false
  fallback_behavior:
    fail: open
- name: MRAP - License Scan
  description: ''
  enabled: true
  policy_scope:
    projects:
      excluding: []
  rules:
  - type: license_finding
    match_on_inclusion_license: true
    license_types:
    - Apple MIT License
    license_states:
    - newly_detected
    branch_type: protected
  actions:
  - type: require_approval
    approvals_required: 1
    role_approvers:
    - developer
  - type: send_bot_message
    enabled: true
  approval_settings:
    block_branch_modification: true
    block_group_branch_modification: true
    prevent_pushing_and_force_pushing: true
    prevent_approval_by_author: true
    prevent_approval_by_commit_author: true
    remove_approvals_with_new_commit: true
    require_password_to_approve: false
  fallback_behavior:
    fail: open
- name: MRAP - any_merge_request
  description: ''
  enabled: true
  policy_scope:
    projects:
      excluding: []
  rules:
  - type: any_merge_request
    branch_type: protected
    commits: unsigned
  actions:
  - type: require_approval
    approvals_required: 1
    role_approvers:
    - developer
  - type: send_bot_message
    enabled: true
  approval_settings:
    block_branch_modification: true
    block_group_branch_modification: true
    prevent_pushing_and_force_pushing: true
    prevent_approval_by_author: true
    prevent_approval_by_commit_author: true
    remove_approvals_with_new_commit: true
    require_password_to_approve: false
  fallback_behavior:
    fail: open
scan_execution_policy:
- name: SEP - Pipeline
  description: ''
  enabled: true
  policy_scope:
    projects:
      excluding: []
  rules:
  - type: pipeline
    branches:
    - "*"
  actions:
  - scan: secret_detection
- name: SEP - Schedule
  description: ''
  enabled: true
  policy_scope:
    projects:
      excluding: []
  rules:
  - type: schedule
    cadence: 0 0 * * *
    branch_type: default
  actions:
  - scan: secret_detection
pipeline_execution_policy:
- name: PEP - Inject
  description: ''
  enabled: true
  pipeline_config_strategy: inject_ci
  content:
    include:
    - project: read-model-migration/read-model-migration-security-policy-project
      file: basic.yml
vulnerability_management_policy:
- name: Vulnerability management policy
  description: ''
  enabled: true
  rules:
  - type: no_longer_detected
    scanners: []
    severity_levels:
    - critical
    - high
  actions:
  - type: auto_resolve

• Create basic.yml (used by the pipeline execution policy) in the policy project with:

stages:
  - .pipeline-policy-pre
  - build
  - custom
  - test
  - .pipeline-policy-post

compliance-job:
  stage: .pipeline-policy-pre
  script:
    - echo "Compliance"
policy build job:
  stage: build
  script:
    - echo "Build"
policy test job:
  stage: test
  script:
    - echo "Test"
policy custom job:
  stage: custom
  script:
    - echo "Custom"

duplicate-job:
  stage: test
  script:
    - echo "Duplicate job"

• Assign this project as the security policy project by going to Secure -> Policies -> Edit Policy Project and select the created project
• Execute the migration by doing bundle exec rake db:migrate
• Verify that the policies are created by doing:

Security::Policy.all.map(&:name)
Security::Policy.all.map(&:projects)
Security::ApprovalPolicyRule.all
Security::ApprovalPolicyRule.all.map(&:projects)
Security::ScanExecutionPolicyRule.all
Security::VulnerabilityManagementPolicyRule.all

Addresses #464033 (closed)