Skip to content

fix: Throttle Pages RPS to bypass rate limits

Nailia Iskhakova requested to merge 9-delay-pages-requests into main

The MR adds delay for Pages requests to throttle RPS. With 1 s delay it's more easy to predict how concurrency will affect the RPS. Here's the example of 10 minutes run with concurrency 15 - no 429 errors 🎉

go run cmbr.go --instance https://pages_test.staging.gitlab.io -concurrency 15 -duration 10m -traffic pages 
INFO[0011] crawling...                                   count=100
INFO[0021] crawling...                                   count=200
INFO[0029] crawling...                                   count=300
INFO[0038] crawling...                                   count=400
...
INFO[0550] crawling...                                   count=5600
INFO[0558] crawling...                                   count=5700
INFO[0566] crawling...                                   count=5800
INFO[0575] crawling...                                   count=5900
INFO[0583] crawling...                                   count=6000
INFO[0591] crawling...                                   count=6100

Also verified that throttle doesn't affect other traffic options.

Related Slack discussion: https://gitlab.slack.com/archives/C1BSEQ138/p1658243931251419?thread_ts=1658236359.724929&cid=C1BSEQ138

Related to #9

Merge request reports