## Release notes Application security scanning is an important part of any secure development process. There are a wide variety of tools and technologies covering the entire SDLC from source code scanning to post-deployment application and infrastructure scanning. While the ultimate goal of any of these tools is to discover both known and potential vulnerabilities, the information coming from any given scanner can vary widely. Efforts to standardize scanning output data do exist but they tend to focus only on one category of scanning technology. Oasis SARIF for static analysis is one such example. This presents a big challenge to security teams who need to aggregate a wide array of scanner findings. Without a consistent way to aggregate and normalize disparate findings, scanner results are often reviewed in the source tool, leaving the true picture of vulnerability risk fragmented and sitting outside of the rest of the DevOps toolchain. The generic security report schema will bridge this gap. Integrate virtually any type of security scanner into GitLab—even home-grown tools—with minimal effort. The schema's structure makes it easy to map most tool's existing outputs into our new JSON format. Flexibility without sacrificing the ability to provide rich vulnerability finding data is a primary purpose behind the new schema. There are only minimal required pieces of information for any finding. The rest of the details are provided in an open structure using pre-defined data types. The pre-defined types handle both data validation as well as standardized UI presentation inside GitLab. For instance, we provide types such as Integer, URL, Table, and even GFM ([GitLab Flavored Markdown](https://docs.gitlab.com/ee/user/markdown.html#gitlab-flavored-markdown-gfm)). This allows granular control over how finding details are presented while keeping the overall experience inside GitLab consistent. ## Purpose A major motivation behind creating a generic schema is our own Common Report Format. The idea of a base set of required fields with extensions is very similar. However, the current Common Report Format is effectively a starting point from which each scanner type has created its own "flavor". For example, there are separate [SAST](https://docs.gitlab.com/ee/user/application_security/sast/#reports-json-format), [DAST](https://docs.gitlab.com/ee/user/application_security/dast/#reports), and [Container Scanning](https://docs.gitlab.com/ee/user/application_security/container_scanning/#reports-json-format) "common" formats that have different incompatible requirements and are separately versioned. It also means vulnerability management must update its parsers every time any of these formats changes. This adds overhead and unnecessary complexity as the majority of fields inside the reports are used only for display purposes. Having scanner-specific formats also complicates integrations, both for 3rd-party vendors as well as any home-grown security scanning tools. You must pick the correct format that most closely matches your scanner's output and then hope you can can reasonably map it to the chosen Common Report. It is also difficult to extend the Common Reports to provide any extra or custom information. Many integrating vendors chose to simply stuff extra information into the `description` and other existing fields. Formatting options are limited so this leads to a less than ideal output—and user experience. With the generic security report schema, we will build on lessons learned with an eye toward these driving philosophies: 1. Schema will explicitly define only what information is needed to make application/business logic decisions. 1. All other data is considered display-only (part of the generic, extensible structure) 1. Individual scanner teams can make exceptions to this by creating logic to pull from the display-only section for their specific scanner's output. However, this logic will live outside the main parser and will not be added as pre-defined fields in the main schema unless broadly required by other scanner teams. 1. We can always make additional fields explicitly defined (and possibly required) in future spec versions. 1. We can use the generic schema for our own scanners as well as virtually any similar 3rd-party tool or even home-grown tools. 1. The generic schema is what vulnerability management parses directly. If we want to support industry formats such as [SARIF](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=sarif), we should look at creating middleware to translate to and from the generic schema. ## Scope Much work has already been done on the generic security report schema in the form of various POCs by @d0c-s4vage. The direction he has taken is in line with what the goals and direction outlined above. We should leverage his MRs and designs as much as possible. At the same time, look for opportunities to make improvements that ensure scalability and production-readiness when this new schema is broadly used across the GitLab user base. ### Related Items To help outline the work done, here are related items and how they roughly fit together: * Primary Issue: [POC: Restructure Security Reports](https://gitlab.com/gitlab-org/gitlab/-/issues/233168) * Primary POC MR: [Draft: Adds generic security report type](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41195) * Supporting POC: [Draft: Adds security report details field rendering](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41783) * Supporting UX Issue: [Design: Templated components for generic report schema](https://gitlab.com/gitlab-org/gitlab/-/issues/267193) * Supporting Implementation: [Introduce `details` attribute for `Vulnerabilities::Finding` model](https://gitlab.com/gitlab-org/gitlab/-/issues/263497) [Draft: Adds generic security report schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/merge_requests/45) may not fully apply anymore as one of its referenced MRs was superseded by gitlab!41195 but may still be helpful for reference. Specifically, the schema of the `details` field contained in the MR could be a base to build on. Also related is work by ~"group::vulnerability research" to [improve vulnerability tracking](https://gitlab.com/groups/gitlab-org/-/epics/4612). Their [research work](https://gitlab.com/gitlab-org/gitlab/-/issues/233167) will eventually allow the scanner teams to improve their fingerprinting. ~"group::threat insights" will need to implement a `tracking` parameter as part of the generic security report format. Each new tracking method will need a corresponding addition and validation. ### Expected Outcomes This Epic delivers: 1. A generic schema in the [security-report-schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas) project. 1. A report validator that can identify incorrectly formatted reports. Ideally, the validator can be used to check outputs outside of pipelines. This will be useful when developing integrations of new scanners/tools. 1. ~backend changes necessary to validate, ingest, and store reports adhering to the new generic schema. 1. Report parser 1. Rails models 1. Database 1. ~frontend changes necessary to render the information on the relevant locations: 1. Security dashboard ([example](https://gitlab.com/gitlab-org/gitlab/-/security/dashboard/?state=DETECTED&state=CONFIRMED)) 1. Vulnerability page ([example](https://gitlab.com/gitlab-org/gitlab/-/security/vulnerabilities/3633816)) 1. Pipeline security tab ([example](https://gitlab.com/gitlab-org/gitlab/-/pipelines/193796036/security)) 1. Pipeline security finding modal (for an example, open example above and click on any finding) 1. MR security widget ([example MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43218) - hit "Expand" on security report sections) 1. MR security widget finding modal (for an example, open example above and click on any finding) See [Design issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233168) for discussions and more details.