DAST vulnerabilities show more information about the request - Backend

This is tracking backend work for #36332 (closed)

Implementation

• The request and response should be added to evidence in the DAST Report Format schema.
• DAST should be converted to generate the JSON report based on accessing the ZAP API, not by using the generated ZAP report. This is because the ZAP report does not contain the information required that lets us obtain the request/response associated with the vulnerability.
• DAST should add the request and response to the evidence in the JSON report.
• DAST tests should be updated to include request and response output. Information that changes with each request should be normalized before comparison in the tests.
• DAST should be released with a new minor version.
• The DAST schema should be released with a new patch version.
• GitLab Rails should be updated to expose the request and response information.
• The request and response should be displayed on the Security Dashboard vulnerabilities, Pipeline Security Tab, Merge Request widget, and new standalone vulnerability page (done in another issue).
• Security products other than DAST should not expose the request/response information.
• Clean up the old ZAP Report usage in the DAST report classes
• Add to 14.0 deprecation building the ZAP report using -J Not considered required.
• Update DAST documentation to remove any -J limitations
• Communicate the changes to the way headers are stored {} -> [{}, {}]
• Ensure that query parameters are added to the URL
• Investigate status code of 0 (see full scan test) (seems to be no different to the known redirect error)
• Use a feature toggle so we can verify the feature is ready (safe) to be released)
• Extract Vulnerabilities from the Secure Report Format generation in DAST Python
• Turn feature toggle off (once approved)
• Expose request/response headers in the report, and for certain headers, the mask the value with 8 asterisks. e.g. Authorization: ********
• Users can set the headers to mask using the environment variable DAST_MASK_HTTP_HEADERS. Multiple values can be supplied if separated by a comma e.g DAST_MASK_HTTP_HEADERS=Authorization,Cookie.
• If no headers are defined by DAST_MASK_HTTP_HEADERS, mask the headers Set-Cookie, Cookie, Proxy-Authorization and Authorization.
• Header names should be case insensitive. e.g. DAST_MASK_HTTP_HEADERS=set-cookie
• Cookie/Set-Cookie headers will be treated as special cases, their values will be masked. e.g. Cookie: sessionId=**********; bizId=**********
• Values that are part of the Set-Cookie header should not be masked, e.g. HttpOnly, Secure. See possible values.
• Document usage of the new variables in the DAST documentation.