E2E Testing

Description

Test and define E2E cases to ensure supported paths don't break.

Preparations

1. Create new root group
2. Create 3 Frameworks: F1, F2 and F3
3. Create 2 Project: project-f1-f2 and project-f1-f3
4. Assign F1 and F2 frameworks to project-f1-f2 and F1 and F3 to project-f1-f3

Test Cases

Test Case	Expected Outcome	Result - Date - Tester (@ + your profile) - Reference	E2E Test
Create new Scan Execution Policy scoped to compliance framework labels F1 and F2.	In both projects when pipeline is executed there is only one job enforced, such that the jobs from the single SEP are de-duplicated.	✅ - @alan - Link to test	Link to E2E test case we've created
Create two Pipeline Execution Policies (Policy Overlap 1, Policy Overlap 2). In Policy Overlap 1, create a basic SAST job. In Policy Overlap 2, create a basic Secret Detection job. Create one compliance framework (Critical). Scope Policy Overlap 1 to projects with the Critical Compliance Framework label Scope Policy Overlap 2 to projects with the Critical Compliance Framework label	If there are two different policies that overlap with the same project, one targeting the critical label and one targeting the high label, those would be two distinct policies and we'd enforce both. Be sure to meet rules/requirements of SAST and SD jobs for them to execute.
Create one Pipeline Execution Policy (Single PEP) with a basic SAST Job. Create two compliance frameworks (Critical, High). Scope Single PEP to projects with the Critical Compliance Framework label Scope Single PEP to projects with the High Compliance Framework label	If a Project has multiple labels, such as Critical and High, if a pipeline execution policy is scoped to projects containing either label, we should only apply the pipeline once. Put another way, we should "de-duplicate" any requirements for enforcement, ensuring the rule is enforced from the same pipeline execution policy without duplicating policy jobs that do the same thing. Only one SAST job should execute. Be sure to meet rules/requirements of SAST for the job to execute.
Create two Pipeline Execution Policies (PEP - Critical, PEP - High). In PEP - Critical, create a basic SAST job. In PEP - High, create a basic Secret Detection job. Create two compliance frameworks (Critical, High). Scope PEP - Critical to projects with the Critical Compliance Framework label Scope PEP - High to projects with the High Compliance Framework label	This is a standard null case where there is no overlap in the jobs and policies, or overlap with framework labels. The SAST job should run for projects with Critical compliance framework label. The SD job should run for projects with High compliance framework label. Jobs should execute in Self-managed instances for all projects that are linked and scoped across all top-level groups. In GitLab.com namespaces, jobs should execute in all linked projects scoped to the label within the same namespace.
Create one Pipeline Execution Policy (Single PEP) with a basic SAST Job. Scope/enforce against a single project "Testing includes with PEP". Add a project include for the project. Have the SAST job defined after the include in the pipeline-execution.yml. Define the policy to use Inject mode and NOT override mode.	When an include is added to a pipeline execution policy using Inject mode, we should not duplicate jobs.
Assign multiple compliance framework labels (e.g., Critical and High) to a project and create two compliance frameworks with compliance pipelines. In the first framework, enforce a basic SAST job. In the 2nd framework, enforce a basic SD job.	Expected outcome is that it will be variable which compliance pipeline takes precedence.
Assign multiple compliance framework labels (e.g., Critical and High) to a project and create one compliance framework (Critical) with a compliance pipeline. Create a pipeline execution policy that is scoped to the High compliance framework. In the compliance pipeline, enforce a basic SAST job. In the pipeline execution policy, enforce a basic SD job.	Expected outcome is that the compliance pipeline will override any PEPs targeting the project, taking full precedence.
Create two merge request approval policies (Override + 1 approval, No override + 0 approvals). Create one framework Critical. In Override + 1 approval create an MRAP that requires 1 approval and deselect all project settings that can be overridden. In No override + 0 approvals create an MRAP that requires no approvals and select all project settings that can be overridden. Scope both policies to projects with the framework Critical.	Expected outcome is that all MRAPs are processed and default to the most secure position. All true settings to override the project are accepted. Approval rules stack such that in this case 1 approval would be required based on the action defined in Override + 1 approval For No override + 0 approvals another security approval rule would be created that requires 0 approvals, so appears optional.
Create all three policy types. Assign to a compliance framework.	The scope is correctly displayed in the policy page for each of the policies. The policies show correctly as scoped to the framework in the compliance center and when editing the framework

To be defined:

1. Verify behavior of multiple labels with Scan Execution Policies -- need more test cases here.
2. Verify behavior of multiple labels with Pipeline Execution Policies and with limits of PEP
3. Advanced configurations of Pipeline Execution Policies (ie. changing strategies: inject vs override).
4. Verify behavior when no .gitlab-ci.yml file exists
5. Behavior when you have all policies applied to these scopes
6. Behavior for custom stages when using Inject mode with pipeline execution policies - video