Group URLs configuration for DAST

While Spike: Experiment for the page similarity algor... (#526761 - closed) • Arpit Gogia • 18.3 identified an algorithmic approach to compare two webpages for similarity, this issue describes a new configuration for DAST such that users can specify a group of URLs that would be considered similar and hence only be crawled once.

Who would this be useful for?

This feature would be useful for websites that have parametric page locations such as:

  • /products/10293/details
  • /products/09u231/details
  • /products/81282/details

Assuming these pages are essentially similarly rendered, with similar HTTP requests and interact-able elements, DAST doesn't need to crawl all 3.

Users will be able to specify DAST_GROUP_PATHS: /products/*/details, ... to avoid redundant crawling

Effect

DAST will crawl the first page that matches the path(s) specified in DAST_GROUP_PATHS and ignore any further encountered pages matching the same pattern

Edited by 🤖 GitLab Bot 🤖