Improve DAST target validation checking

Problem to solve

Before DAST scans a website, the target website is checked to verify that it is up and running. This is to ensure that the ZAP scan is able to spider all URLs on the website, and helps ensure results are deterministic.

Unfortunately, some websites have target URLs which cause problems with the check. These include:

• Websites that take longer than 5 seconds to load
• Websites that have lots of redirects
• Websites that return status codes in the range 400-500

Intended users

• Sasha (Software Developer)
• Sam (Security Analyst)

Implementation plan

• DAST should continue to scan when the website returns a 400 series error. The error is a client error, which indicates that the server is up and running.
• DAST should provide an environment variable, DAST_SKIP_TARGET_CHECK. When set to true, the target check made by DAST should be skipped entirely. DAST_SKIP_TARGET_CHECK should be false by default.
• More information should be emitted to explain why the website check failed. See #237842 (closed) for a suggestion on how this might look. See also #245255 (closed) for an explanation on what information would be useful.
• If the site check fails, the job log should include a message to explain to the user what options they have, for example, "Set the variable DAST_SKIP_TARGET_CHECK to false to prevent this check".

Documentation

Please document the new environment variable in the DAST documentation.

Availability & Testing

What is the type of buyer?

Gold/Ultimate