Secure testing gap analysis map

This issue shall map features present in Secure against current testing resources.

Any gaps should arise and be raised/triaged for action accordingly.

For the purposes of this ticket, the following are defined:

• Unit test, typically under /specs, testing code functions

• Integration test could either be tests under /specs that test the integration of functions, or tests of projects under the Security Test projects https://gitlab.com/gitlab-org/security-products/tests/

• E2E test - typically UI driven and live under https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/specs/features/ee/browser_ui/secure

Scanners

Scanner	Unit test	Integration test	E2E test
Container Scanning - PROTECT STAGE
Threat Monitoring
Dependency Scanning

Threat Monitoring

Feature	Unit test	Integration test	E2E test
Container Network Policy	✅ cluster application; auto-deploy-app; backend models: network policy cilium network policy; backend controller; backend services; frontend: policy editor, network policy list, network policy drawer, threat monitoring sections

Dependency Scanning

Feature	Unit test (see matching language/framework in dir)	Integration test (see *-qa jobs in linked ci files)	E2E test
Ruby Bundler	✅ link	✅ link
PHP Composer	✅ link	✅ link
C Conan	✅ link	✅ link
C++ Conan	✅ link	✅ link
Go Golang	✅ link	✅ link
Maven	✅ link	✅ link
Gradle	✅ link	✅ link
SBT	✅ link	✅ link
Javascript NPM	✅ link	✅ link
.NET nuget	✅ link	✅ link
Javascript Yarn	✅ link	✅ link
C# nuget	✅ link	✅ link
Python setuptools	❌	❌
Python pip gitlab-org/quality/testcases#1091 (closed)	✅ link	✅ link
Python pipenv gitlab-org/quality/testcases#1091 (closed)	✅ link	✅ link
Python poetry gitlab-org/gitlab#7006 (closed)	✅ link	❌ gitlab-org/gitlab#299294
Maven (test building java 8 to 14)	❌	✅ link
Gradle (test building java 8 to 14)	❌	✅ link
Scala SBT (test building java 8 to 14)	❌	✅ link
Javascript NPM using retire.js scanner	❌	✅ link
Ruby Bundler using bundler-audit scanner	❌	✅ link
Environment Variables gitlab-org/gitlab#294208 (closed) , gitlab-org/gitlab#294207 (closed)	❌	❌
Omnibus (GitLab only) with gitlab-depscan	✅ link	❌
Offline java projects (mvn, gradle, scala)	❌	✅ (see offline-FREEZE branch in java-maven, java-gradle, scala-sbt test projects)
Offline python projects (pip, pipenv, setuptools	❌	✅ (see offline-FREEZE branch in python-pip, python-pipenv test projects)
Offline js project (npm, yarn)	❌	✅ (see offline-FREEZE branch in js-npm, js-yarn test projects)
Offline ruby project (bundler)	❌	✅ (see offline-FREEZE branch in rub-bundler test project)
Private mvn repo	❌	❌
Private pypi repo	❌	❌
Private npm repo	❌	❌
Private gem repo	❌	❌
Remediation	❌	❌	✅ link
Enable within the UI ( gitlab-org/gitlab#284108 (closed) )			❌ gitlab-org/quality/testcases#1666 (closed)
Use a different stage (instead of default "test")			❌

Dependency List

Feature	Unit test	Integration test	E2E test
Project level list	✅ frontend controller parser	frontend: ❌ backend: ❌	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/ee/page/project/secure/dependency_list.rb
Click arrow to view dependencies	✅ frontend no need for backend test	frontend: ❌ backend: ❌
Dependency Paths	✅ frontend, frontend, backend	frontend: ❌ backend: ❌
Licenses are shown in same list	✅ frontend controller	frontend: ❌ backend: ❌
Empty state UI	✅ frontend, frontend controller	frontend: ❌ backend: ❌	gitlab-org/quality/testcases#1128 (closed)
Vulnerablities related to dependency	✅ controller parser	backend: ❌

SAST

Feature	Unit test	Integration test	E2E test
Project level list
.NET Core
.NET Framework
Apex Salesforce
C/C++
Elixir
Go
Groovy Ant
Groovy Gradle
Groovy Maven
Groovy SBT
Helm Charts
Java Ant
Java Gradle
Java Maven
Java SBT
Java Android
JS
Kotlin Android
K8s manifests
NodeJS
ObjectiveC iOS
PHP
Python PIP
React
Ruby on Rails
Scala Ant
Scala Gradle
Scala Maven
Scala SBT
Swift iOS
Typescript
Tier support (CE/Core vs EE/Ultimate)
Environment variables
Kubesec
Offline
bandit
brakeman
eslint
flawfinder
gosec
kubesec
mobsf
nodejs-scan
phpcs-security-audit
pmd-apex
security-code-scan
Enable via UI	✅ component specs	🔶 weak frontend	✅ gitlab-org/quality/testcases#1667 (closed)
Use a different stage (instead of default "test")			❌
False positives https://gitlab.com/gitlab-org/gitlab/-/issues/292686			❌ gitlab-org/quality/testcases#2259 (closed)

Secret Detection

Feature	Unit test	Integration test	E2E test
Gitleaks ruleset https://docs.gitlab.com/ee/user/application_security/secret_detection/#supported-secrets
Environment variables including Full History Secret Scan
Custom ruleset
Offline
Use a different stage (instead of default "test")			❌

Auto Remediation

Feature	Unit test	Integration test	E2E test
Create remediation MR	✅ backend; frontend - not applicable	❌	❌
Show solutions in Vulnerability report	Frontend: UI-heavy component, covered by the integration tests; backend	✅ frontend Link ❌ backend	❌
Settings	✅ Frontend: UI-heavy component, covered by the integration tests; backend	✅ frontend Link ❌ backend	❌
Available MRs in Project Security Dashboard header	✅ Frontend: UI-heavy component, covered by the integration tests; backend not applicable	✅ frontend Link ❌ backend	❌

DAST

Feature	Unit test	Integration test	E2E test
Project level View details of a DAST vulnerability
Environment Variables
Full / API / URL scans
DAST on demand scans	✅ Frontend		gitlab-org/quality/testcases#1122
DAST profiles	✅ Frontend		gitlab-org/quality/testcases#1122
Use a different stage (instead of default "dast")			❌

API Fuzzing

Feature	Unit test	Integration test	E2E test
Fuzzing scan
OpenAPI
HTTP Archive
Postman
Environment Variables
Overrides
Header fuzzing
Turn off a check/assertion
Offline
Vulnerability MR widget	✅ Frontend
Vulnerability Detail Page	✅ Frontend
Pipeline Security Tab	✅ Frontend
Security Dashboard	✅ Frontend
Artifacts Download	✅ Frontend
Artifacts
Use a different stage (instead of default "fuzz")			❌

Coverage fuzz testing

Feature	Unit test	Integration test	E2E test
C/C++
GoLang
Swift
Rust
Java JQF
Java javafuzz
Full fuzz
Regression fuzz
Environment variables
Offline
Continuous Fuzzing
Vulnerability MR widget	✅ Frontend
Vulnerability Detail Page	✅ Frontend
Security Dashboard	✅ Frontend
Pipeline Security Tab	✅ Frontend
Fuzzing artifacts	✅ Frontend
Artifacts
Use a different stage (instead of default "fuzz")			❌

License Compliance

License Compliance

Feature	Unit test	Integration test	E2E test
In MR	✅ frontend specs ✅ backend	🔶 weak frontend ❌ backend	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
In Pipeline	✅ frontend specs (same as MR frontend specs above) backend	frontend, backend: ❌	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb
Approve license	frontend spec, frontend spec backend	frontend, backend: ❌	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
Deny license	frontend spec, frontend spec backend	frontend, backend: ❌	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
Javascript Bower	no test	✅ link
Javascript NPM	no test	✅ link
JavaScript Yarn	✅	✅ link
Go Godep	No test	❌
Go mod	No test	✅ link
Java Gradle	no test	✅ link
Java Maven	✅ link	✅ link
.NET nuget	✅ link	✅ link
Python Pip	✅ link	✅ link
Ruby Gem	✅ link	✅ link
Experimental supported languages	only yarn	rust, Pipenv
Environment variables	❌	❌
Custom Dependencies	❌	❌
License Approvals within a project	frontend specs, frontend specs	frontend: ❌
License policy tab gitlab-org/gitlab#219461 (comment 446528615)	frontend spec	frontend: ❌
Empty state dashboard	frontend specs	frontend: ❌	✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb#L33
Use a different stage (instead of default "test")			❌

Dashboards / UI driven

Feature	Unit test	Integration test	E2E test
Project level View details of a DAST vulnerability
MR View details of a DAST vulnerability			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
MR View details of a Dependency Scanning vulnerability			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
MR View details of a Container Scanning vulnerability			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
Dismiss a vulnerability			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Dismiss multiple vulnerabilities
Create an issue for a vulnerability (nicole assumes w/out edits)			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Create an issue for a vulnerability WITH EDITS			🚧 gitlab-org/quality/testcases#1140 (closed)
New create a JIRA issue for a vulnerability			gitlab-org/quality/testcases#1101 (closed)
Automatic remediation for vulnerabilities - DS	✅ see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#auto-remediation		✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Automatic remediation for vulnerabilities - CS	❌
Automatic remediation for vulnerabilities - Manually apply the suggested patch
Create MR from a vulnerability
Create MR from a vulnerability twice, check message
Add related issue for a vulnerability
Remove related issue for a vulnerability
Security approvals in MRs
License Approvals within a project	see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#license-compliance
Security report out of date
Security Configuration status
Security Configuration Autodevops
Security Configuration SAST
Security Configuration DAST
Dependency List	see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#dependency-list		✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb#L133
Standalone vulnerabilities on Dependency List page			❌ gitlab-org/quality/testcases#1672 (closed)
Security Dashboard Secrets
Pipeline Security tab Secrets
MR widget Secrets
Pipeline Security Dashboard			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Project Security Dashboard graph gitlab-org/quality/testcases#1092
Project Vulnerability Report			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/project_security_dashboard_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Group Security Dashboard			✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Security Centre
Export
Vulnerability Report
Project Security Dashboard when pipeline is blocked by a manual step (gitlab-org/gitlab#247490 (closed))			❌

Other?

Feature

Feature	Unit test	Integration test	E2E test
CI YAML templates present/not?
CI for MRs
Autodevops
Feature flag enable/disable MR security widget	frontend specs
Change vulnerability status

Schemas

https://gitlab.com/gitlab-org/security-products/security-report-schemas/

Likely E2E candidates:

Feature	Unit test	Integration test	E2E test
Test against minimal fields returned
Test against all possible fields returned

Dashboard

Feature	Unit test	Integration test	E2E test
Dismiss vulnerability
Create issue
Automatic Remediation

Offline

Feature	Unit test	Integration test	E2E test
Private Maven repo
Fuzzing

Out of Scope (Protect Stage)

Container Scanning

Feature	Unit test	Integration test	E2E test
Environment variables
Vulnerability allowlisting
Offline