Customers can view a crawl report from a particular URL

Problem

Sometimes a scan runs and misses specific pages that the customer wants to scan. At the moment, it is hard for them to understand why the page was skipped.

Proposal

The authentication report is a useful means of helping customers understand why a scan isn't authenticating. This issue proposes that a similar report can be produced for any URL in the scan.

• The customer can define a URL of interest
• The first time the crawler performs an action that results in that URL, an observer starts recording requests
• All requests and responses from that navigation are recorded
• The new navigations are recorded
• The observer stops recording requests once the navigation is complete
• All information is outputted in a single HTML report that can be exported as an artifact (similar to the authentication report)

Potential problems

Performance for a really large artifact file may be a problem.

Reference

This is an alternative to Customers should be able to view the browser of... (#362721).