Skip to content

feat: Introduce Pages only crawling to CMBR

Nailia Iskhakova requested to merge 7-pages-traffic into main

The MR adds ability to loop through the Pages to generate traffic. Unfortunately it appears there is no way to pull existing project pages URLs from instance. Instead we enable AllowURLRevisit for CMBR in pages mode to be able to loop through the subset of pages. See #7 (comment 1012550283) for more information.

For Staging this group was created https://staging.gitlab.com/pages_test and populated with project pages using steps listed in new docs for pages crawling

Example run
go run cmbr.go --instance https://pages_test.staging.gitlab.io -concurrency 8 -duration 2m -traffic pages
INFO[0004] crawling...                                   count=100
INFO[0006] crawling...                                   count=200
INFO[0009] crawling...                                   count=300
INFO[0014] crawling...                                   count=400
INFO[0017] crawling...                                   count=500
INFO[0020] crawling...                                   count=600
INFO[0023] crawling...                                   count=700
INFO[0027] crawling...                                   count=800

Screenshot_2022-06-30_at_21.44.35

Closes #7 (closed)

Edited by Nailia Iskhakova

Merge request reports