Migrate DAST tool to the common report format
Problem to solve
DAST tool doesn't use our common library JSON report format but uses ZAProxy JSON report format. This prevents us:
- from utilizing a generic API to the report contents in our UI
- from properly documenting the report format for DAST in a way that is consistent with other tools of ours
Intended users
Any user of the Security Products within GitLab
Further details
The DAST tool is migrating from a custom ZAProxy format to use the Common Report Format. This will enable the Secure team to reuse key parts of our codebase.
The migration has three stages:
- DAST will create a
gl-dast-report.jsonreport containing the legacy ZAProxy fields and the Common Report Fields. #14053 (closed) - The GitLab Rails codebase will use the Common Report Format for DAST instead of the ZAProxy fields. #33913 (closed)
- DAST will remove the legacy ZAProxy fields from the report. #33915 (closed)
This issue represents step 1. of the migration.
Proposal
Note: The DAST Architecture Migration has recently been updated to create the Common Report Format in Python, as opposed to using the Common Project.
The gl-dast-report.json report produced by DAST has the following top level fields:
{
"@version": "D-2019-09-23",
"@generated": "Tue, 1 Oct 2019 04:16:28",
"site": [...],
"spider": {...}
}The Common Report Format has the following top level fields:
{
"version": "2.2",
"target_url": "http://...",
"vulnerabilities": [...],
"scanned_urls": {...},
"io_error_urls": {...}
}Producing a file that contains both these fields will allow DAST to be upgraded to use the Common Report Format and give the GitLab codebase time to transition away from the Legacy DAST format.
Specifically, this issue should
- Using Python, map ZAP output to the Common Report Format (see below),
- Add these values with the normal ZAP output to the
gl-dast-report.json - Convert the
ee/spec/fixtures/security_reports/master/gl-dast-report.jsonin the GitLab codebase to include the added extra fields. Run tests, ensure the added fields don't break GitLab.com.
An example: current DAST output
{
"@version": "D-2019-09-23",
"@generated": "Tue, 1 Oct 2019 04:16:28",
"site": [
{
"@name": "http://nginx",
"@host": "nginx",
"@port": "80",
"@ssl": "false",
"alerts": [
{
"pluginid": "10016",
"alert": "Web Browser XSS Protection Not Enabled",
"name": "Web Browser XSS Protection Not Enabled",
"riskcode": "1",
"confidence": "2",
"riskdesc": "Low (Medium)",
"desc": "<p>Web Browser XSS Protection is not enabled, or is disabled by the configuration of the 'X-XSS-Protection' HTTP response header on the web server</p>",
"instances": [
{
"uri": "http://nginx/",
"method": "GET",
"param": "X-XSS-Protection"
}
],
"count": "6",
"solution": "<p>Ensure that the web browser's XSS filter is enabled...</p>",
"otherinfo": "<p>The X-XSS-Protection HTTP response header allows...</p>",
"reference": "<p>https://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet</p><p>...</p>",
"cweid": "933",
"wascid": "14",
"sourceid": "3"
}
]
}
],
"spider": {
"progress": "100",
"state": "FINISHED",
"result": {
"urlsInScope": [
{
"processed": "true",
"statusReason": "OK",
"method": "GET",
"reasonNotProcessed": "",
"url": "http://nginx/",
"statusCode": "200"
}
],
"urlsOutOfScope": [],
"urlsIoError": []
}
}
}Mapping to the Common Report Format
The following shows the current Common Report Format, and the mapping to this from the ZAP output. Note that every instance of an alert ends up as a separate Common Report Format vulnerability.
{
"version": "#latest version of Common Report Format at time of building",
"vulnerabilities": [
{
"category": "dast",
"confidence": "#vulnerability[].confidence, mapped",
"cve": "#vulnerability[].pluginid",
"description": "#vulnerability[].desc, html removed",
"identifiers": [
{
"name": "#vulnerability[].name",
"type": "ZAProxy_PluginId",
"url": "https://github.com/zaproxy/zaproxy/blob/w2019-01-14/docs/scanners.md",
"value": "#vulnerability[].pluginid"
},
{
"name": "CWE-#vulnerability.cweid, this object only present if cweid exists",
"type": "CWE",
"url": "https://cwe.mitre.org/data/definitions/#vulnerability.cweid.html",
"value": "#vulnerability.cweid"
},
{
"name": "WASC-#vulnerability.wascid, this object only present if wascid exists",
"type": "WASC",
"url": "http://projects.webappsec.org/w/page/13246974/Threat%20Classification%20Reference%20Grid, subpage, mapped based on #vulnerability.wascid",
"value": "#vulnerability.wascid"
}
],
"links": [{
"url": "#vulnerability[].references[].url, html removed"
}],
"location": {
"param": "#vulnerability[].instances[].param",
"method": "#vulnerability[].instances[].method",
"hostname": "#vulnerability[].instances[].uri, only scheme://userinfo@hostname:port portion",
"path": "#vulnerability[].instances[].uri, only /path?query#fragment portion"
},
"message": "#vulnerability[].name",
"scanner": {
"id": "zaproxy",
"name": "ZAProxy"
},
"severity": "#vulnerability[].riskcode, mapped",
"solution": "#vulnerability[].solution, html removed"
}
]
}Out of scope
The fields for evidence/request/response are part of a larger discussion, so have been removed from the scope of this story. See #37027 (closed), #36332 (closed).
The fields for representing scanned_urls are part of a larger discussion about whether or not to include a scan/scans element in the Common Report Format, hence are also out of scope. See #11024 (closed).
Documentation
Resolving this issue will unlock the documenting of the DAST report format.
Testing
Integration tests' fixtures for DAST project will have to be updated according to the new report format.
What does success look like, and how can we measure that?
DAST tool emits a report file in both the common library JSON report format and the DAST ZAProxy format
What is the type of buyer?
GitLab Ultimate users
Links / references
Issues related to moving DAST reports logic to the backend: they define places in the codebase that will require an update to consume common reports format in for DAST reports.