Reliable gitlab-qa smoke tests on staging and the canary stage for deployments

We currently put gitlab-qa smoke tests into our deployment pipeline but lately it has slowed down getting updates to production. This issue is to discuss some improvements we can make to make the QA smoke tests more reliable.

• Too much data under gitlab-qa-sandbox-group is causing smoke tests to fail https://gitlab.slack.com/archives/C3JJET4Q6/p1555082015009400 Can this be avoided with janitor cleanup?
• The password for gitlab-qa-bot might have been changed https://gitlab.slack.com/archives/C3JJET4Q6/p1555079896006200