Approach to E2E tests for the Secure stage functionality of GitLab
Currently, we don't have any system or E2E tests for the functionality of the ~"devops:secure" stage pages (Group Security Dashboard, Project Security Dashboard, Merge Request Security Reports, Pipeline Security Reports, Dependency List).
This issue is to discuss possible approaches from the perspective of the velocity/quality balance and to agree on the right approach.
Secure test plan
Introduction
This is a test plan to backfill basic e2e tests for the current secure features:
-
Merge request security reports | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14847 -
Group security dashboard | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14874 -
Auto-remediation process | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14945 -
Project security dashboard | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14654 -
Pipeline security reports | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/12175 -
Dependency List | MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14876
The goal is to check that the reports are shown for each configured feature after triggering a pipeline that has security products feature enabled. Additionally, we should verify that auto-remediation works to remove vulnerabilities from the merge request.
Scope
- This plan covers ensuring that the various reports and dashboards above are accessible
ACC Matrix
Secure | Responsive | Intuitive | Reliable | |
---|---|---|---|---|
Groups | 2 | |||
Project | 2 | |||
MRs | 1 | 1 | 3 | |
CI/CD | 2 |
Capabilities
- Merge request is
- Intuitive
- When dependency scanning reveals vulnerabilities we can auto-remediate, it should be easy to trigger the auto-remediation process
- Responsive
- When the auto-remediation process is triggered, it should create the patch file immediately
- Reliable
- Once the pipeline for a merge request commit finishes, the merge request security report should be available
- The merge request security report should be accurate
- When the auto-remediation process is triggered and the pipeline finishes, the vulnerabilities should be gone
- Intuitive
- Group is
- Reliable
- Once the pipeline for a merge request commit finishes, the group security dashboard should be available
- The group security dashboard should be accurate
- Reliable
- Project is
- Reliable
- Once the pipeline for a merge request commit finishes, the project security dashboard should be available
- The project security dashboard should be accurate
- Reliable
- CI/CD is
- Reliable
- Once the pipeline for a merge request commit finishes, the pipeline security report should be available
- The pipeline security report should be accurate
- Reliable
Test Plan
Automated end-to-end tests
Scenario 1 Viewing a merge request security report
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, check the merge request security report is present and correct
Scenario 2 Viewing the group security dashboard
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, check the group security dashboard is updated and correct
Scenario 3 Triggering auto-remediation
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, trigger auto-remediation for the merge request
- Once the pipeline finishes, check the merge request security report is present and correct
Scenario 4 Viewing the project security dashboard
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, check the project security dashboard is updated and correct
Scenario 5 Viewing a pipeline security report
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, check the pipeline security report is present and correct
Scenario 6 Viewing the Dependency List
- Go to a project
- Trigger a pipeline that has the security products feature enabled
- Once the pipeline finishes, check the Dependency List is updated and correct
/cc @gl-quality This is my first test plan using the ACC format, so whatever feedback you have would be welcome!