Further improvements for Staging crawler
Follow up on #338978 (closed) to further improve webcrawler setup on Staging to emulate user activity:
-
Switch to use admin user specifically for cmbr
(depends on https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/12758) -
Investigate if we can authenticate for web requests - gitlab-com/gl-infra/cmbr!5 (merged) -
Explore skipping gitlab-qa-sandbox-group
groups and projects as they are being removed quickly and muddy the logs with 404 errors - gitlab-com/gl-infra/cmbr!5 (merged) -
Update CI config - add check jobs for merge requests in cmbr
- gitlab-com/gl-infra/cmbr!5 (merged) -
Break up the tool to make the code cleaner - gitlab-com/gl-infra/cmbr!8 (merged) -
Consider different name for crawler - #349074 (comment 822652871) -
Explore if pipelines need to be moved to ops.gitlab.com
- not needed per #349074 (comment 819091117) -
Investigate Gitaly saturation mentioned in #338978 (comment 748749746) - #349074 (comment 824976353)
Edited by Nailia Iskhakova