DAST degrades when communication with Chromium is overwhelmed
Browser-based DAST uses a headless Chromium instance to crawl the target website. Chromium's events are received internally on a buffered Go channel with a fixed size: https://github.com/wirepair/gcd/blob/master/v2/chrome_target.go#L151
On certain large or complex websites that have a lot of ongoing activity (e.g. requests, DOM manipulations, page loads), the channel can get overwhelmed due to the influx of events. When the channel is completely filled, browser-based DAST will stop receiving messages until space has been made by processing events. In the meantime, some events from Chromium can be lost. This can have unpredictable effects, including unexpected slow downs, crawl coverage being reduced, certain requests/responses not being scanned for vulnerabilities, and even the scan failing entirely (for example if a timeout occurs during auth).
This can appear in logs as:
- Network requests getting missed
- Navigations failing due to a timeout
A straightforward solution is to increase the size of the buffered channel. This will of course only help till we encounter another website that breaches this limit.
Longer term solutions include:
- Processing messages in parallel
- Needs research to determine how messages can be segmented safely, e.g. by request ID.
- Using backpressure to slow down the crawl as the queue gets bigger
- Won't help the current cases as we have seen this happen during the auth phase, but after increasing the size of the buffer it might be more relevant.
- Using a variable-sized buffer
- Not built into go, may have to implement ourselves.
- Reducing time spent handling messages