### Proposal Identify nearly identical pages/blocks of content so they can be skipped to reduce DAST crawl time. Many websites have multiple pages that are nearly identical aside from some text and photos. For example, if DAST were crawling LinkedIn, profile pages would largely contain the same content so it would only be necessary to scan one, rather than 1 million+. ### **Alert** :rotating_light: This piece of work would most likely impact https://gitlab.com/gitlab-org/gitlab/-/issues/522983+s because any change on how we identify duplicate pages after the dark launch has launched could impact our dark launch results. ### [FY26 Roadmap Item](https://docs.google.com/presentation/d/1ABoGLJkQZNs3Y92NELNrRvjsbo_PNEjGMyCRVz2sU2A/edit#slide=id.g3321f028e83_21_104) | Roadmap Issue | Deliverable | Status | |---------------|-------------|--------| | https://gitlab.com/gitlab-org/gitlab/-/issues/555583+s | ~Deliverable | Completed in 18.5 | | https://gitlab.com/gitlab-org/gitlab/-/issues/555581+s | ~Stretch | Planned for 18.5 | ### Implementation Plan The challenge here is comparison of similarity of multiple webpages in an efficient manner i.e. if `/page/1` is similar to `/page/2` within a margin of error, they should be considered similar #### Update 1. After experimenting with SimHash in the spike issue ([notes](https://gitlab.com/gitlab-org/gitlab/-/issues/526761#note_2613285366)), the finalised implementation was a grouped URLs configuration variable. This would allow customers to setup grouped URLs using a wildcard syntax, and the crawler would consider all URLs matching a group to be duplicates. 2. With the configuration released, we will be building upon this configuration and dark launch the SimHash algorithm. 3. The SimHash algorithm would run independently for URLs under each group and a similarity score threshold will be calculated and pushed as a scan metric. 4. This will help us decide on the feasibility of the SimHash algorithm and set a default threshold for the similarity score. ##### Proposal 1. Use a combination of [Locality Sensitive Hashing](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) and an indexing mechanism like a [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) to quickly respond to the question "Is this page similar to any of the 10 other pages seen by the crawler in the past?" 2. Locality Sensitive Hashing will help convert the webpage content to a hash in a manner such that similar content will lead to similar hashes. 3. An index (like Bloom Filter) will keep track of the hashes seen before and their similarity. 4. This comparison can be performed within the crawler after each navigation is executed and its result is available. ##### Proof of Concept https://gitlab.com/-/snippets/3698566 In this PoC, I calculate a locality sensitive hash using [TLSH](https://github.com/glaslos/tlsh), of two GitHub profiles. The page structure is similar with the content being slightly different. Comparing just the "content" parts of the hash (hence the slice from the 4th index in the code), the hashes are very similar:  ### Dependencies * Team dependencies: none * Epic/issue dependencies: none ### DRIs * Engineer: TBD * EM: @mikeeddington (acting) * PM: @joelpatterson * Engineering Owner: @rvider