fix: Add instance availability check
What
Add instance availability check to ensure that CMBR fails early if instance is not available. Step can be skipped by setting environment variable.
Similar to !235 (merged).
Additionally:
- update regex for skip URLs to include more sandbox QA groups and users. These are sandbox groups and have usually broken data, see example https://staging.gitlab.com/gitlab-com/gl-infra/cmbr-staging-load-generator/-/jobs/62702354
- Also increasing timeout for response wait to 60s, default 10s is too short. Some GitLab pages are known to be slow.
- Document that
GITLAB_USER_AGENTis required for Staging crawling
Why
We need to check that target environment is available before running crawler in full, but ensure that any random error on one of the crawler requests doesn't fail the whole pipeline. Flaky errors can exist on env, especially it's seen on Staging.
Resoling issues like ones mentioned in !235 (merged)
We were noticing that while some jobs like this one exit as expected (exit code 1, immediately) upon encountering an error, some jobs like this one that has an invalid url (instance wasn't available) would output an error code, but exit with exit code 0, after sleeping for the set duration.