Before DAST scans a website, the target website is checked to verify that it is up and running. This is to ensure that the ZAP scan is able to spider all URLs on the website, and helps ensure results are deterministic.
Unfortunately, some websites have target URLs which cause problems with the check. These include:
Websites that take longer than 5 seconds to load
Websites that have lots of redirects
Websites that return status codes in the range 400-500
DAST should continue to scan when the website returns a 400 series error. The error is a client error, which indicates that the server is up and running.
DAST should provide an environment variable, DAST_SKIP_TARGET_CHECK. When set to true, the target check made by DAST should be skipped entirely. DAST_SKIP_TARGET_CHECK should be false by default.
More information should be emitted to explain why the website check failed. See #237842 (closed) for a suggestion on how this might look. See also #245255 (closed) for an explanation on what information would be useful.
If the site check fails, the job log should include a message to explain to the user what options they have, for example, "Set the variable DAST_SKIP_TARGET_CHECK to false to prevent this check".
Documentation
Please document the new environment variable in the DAST documentation.
Avielle Wolfemarked the checklist item More information should be emitted to explain why the website check failed. See #237842 (closed) for a suggestion on how this might look. See also #245255 (closed) for an explanation on what information would be useful. as completed
marked the checklist item More information should be emitted to explain why the website check failed. See #237842 (closed) for a suggestion on how this might look. See also #245255 (closed) for an explanation on what information would be useful. as completed
Currently, DAST checks the target twice. The first check is functionality that we built. It never exits the scan. The second check uses ZAP, and was migrated from the ZAP scripts. It exits if ZAP cannot access the target. We don't control that exit, so we can't tell DAST to continue scanning if it receives a 400
Ultimately, we should remove the ZAP target check. There's no point in duplicating it. However, removing it will unexpectedly change behavior for some users, because ZAP accesses the target as provided (not reset to host) and adds the accessed URL to the attack surface. In cases where the target is later reset to host before spidering, the original target might not show up in the spider but still gets scanned because it got added to the attack surface as part of the access check.
Because of this limitation, I plan to work on #257963 (closed) before completing this issue. But since this might mean that some non-host targets will no longer be scanned, I think I should complete #258805 (closed) first.
@avielle It makes sense to remove the ZAP check. This is inline with moving functionality to the python scripts and relying less on ZAP for this type of check.
I'm not clear on how the behavior changes for users or whether users will notice any impact.
@sethgitlab it would change for users who currently have a target that is not a host. For example, the target we use for webgoat is http://localhost/WebGoat/attack. This gets reset to http://localhost before the spider. If the spider never reaches http://localhost/WebGoat/attack (if the target was an un-linked admin page for example), then http://localhost/WebGoat/attack will never be scanned
Right now, the theoretical admin page would be scanned because it gets added to attack surface by the access check. Remove the access check, and it will no longer be included
Avielle Wolfemarked the checklist item DAST should continue to scan when the website returns a 400 series error. The error is a client error, which indicates that the server is up and running. as completed
marked the checklist item DAST should continue to scan when the website returns a 400 series error. The error is a client error, which indicates that the server is up and running. as completed
Avielle Wolfemarked the checklist item DAST should provide an environment variable, DAST_SKIP_TARGET_CHECK. When set to true, the target check made by DAST should be skipped entirely. DAST_SKIP_TARGET_CHECK should be false by default. as completed
marked the checklist item DAST should provide an environment variable, DAST_SKIP_TARGET_CHECK. When set to true, the target check made by DAST should be skipped entirely. DAST_SKIP_TARGET_CHECK should be false by default. as completed
Avielle Wolfemarked the checklist item If the site check fails, the job log should include a message to explain to the user what options they have, for example, "Set the variable DAST_SKIP_TARGET_CHECK to false to prevent this check". as completed
marked the checklist item If the site check fails, the job log should include a message to explain to the user what options they have, for example, "Set the variable DAST_SKIP_TARGET_CHECK to false to prevent this check". as completed