v1: Enable load testing in staging-canary using webcrawler
Part of plan in &6401
Needs #338973 (closed) before starting
Goal: Emulate activity on Staging by using load tool. This should help to increase traffic on Staging and set up alerts on increased error rate using SLI. Infrastructure team uses Grafana dashboards for SLI monitoring.
Task
-
Familiarize with the cmbr
webcrawler and other similar tools. -
Analyse if cmbr
caused intermittent 500 errors in GitLab QA and if so, explore if it's possible to reduce its load further untilstg-cny
is built. - #338978 (comment 752263239) -
Tweak cmbr
to resolve 429 errors - gitlab-com/gl-infra/cmbr!4 (merged) -
Set up webcrawler for Staging & Staging-Canary -
Validate that GitLab QA pipelines are not affected - #338978 (comment 752263239) -
Verify that SLI monitoring for Staging & Staging-Canary is set up - gitlab-com/gl-infra/scalability#1477 (closed) -
Support custom cookies in cmbr
- gitlab-com/gl-infra/cmbr!7 (merged) -
Follow up for further crawler improvements - #349074 (closed)
Resolution: #338978 (comment 822637515)
Future Iteration
- Quality related work &7320 (closed) to streamline load simulation efforts
- Infra team related work: gitlab-com/gl-infra&668 to improve the precision of staging service-level monitoring alerts and set up incident review processes
cc @amyphillips @vincywilson @zeffmorgan @pguinoiseau @ibaum
Edited by Vincy Wilson