DAST should not block EventStream requests

Problem

EventStream requests, a.k.a. Server-Sent Events, will fail to load during a DAST scan. The effect of this failure depends on how the EventStream is being used. It may be relatively benign and have little impact, or it may cause the site to fail to load, resulting in an incomplete scan. We have observed the latter in several customers who are using LaunchDarkly's streaming capability: when the LaunchDarkly state fails to load, the site does not load properly, and the resulting scan is mostly empty.

Context

What is EventStream?

EventStreaming is a formalized variation of the HTTP Long Polling technique which holds the connection open so that the client can receive continuous updates from the server without needing to send a new request. This means that an EventStream response never completes.

Analysis

Because the response from an EventStream never completes, the scanner is unable to get the response body from Chromium. This happens during the Fetch.requestPaused event, and after the attempt to fetch the body times out, the event is never "continued" the way it is supposed to be. This causes Chromium to hang and eventually abort the request.

Known Usages

LaunchDarkly is a hosted feature flag management service which uses EventStream to push flag updates to consuming sites. We have seen multiple customers with failed scans because they were using LaunchDarkly.

Proposal

When handling Fetch.requestPaused for a response, if there is a Content-Type header of text/event-stream, treat the response as if it has no body. We already have similar handling for e.g. redirect responses. This will unblock target sites with an EventStream.

We would want to open a separate issue for actually scanning the messages sent over the EventStream.

Implementation plan

  1. Add needed code change to exclude Content-Type header of text/event-stream from scans
  2. Add new integration tests for the code changes

Testing

See MDN for an example (in PHP!) of a simple script which implements an EventStream on the server. This can be the basis of an end-to-end test which shows that an EventStream does not block the scan.

Edited by Hannah Baker