Consider Renaming CustomersDot Reliable and Sanity E2E Tests to Smoke Tests

Problem

Today, we currently have the following reliable/sanity E2E test suites in CustomersDot:

• qa-only:stg:saas-reliable - blocking, runs in CustomersDot MRs which update the /qa directory
• e2e:saas-reliable - blocking, runs after each CustomersDot Staging deployment
• e2e:production-sanity-check - non-blocking, runs after each CustomersDot Production deployment

We should consider renaming these test suites to be more closely aligned with the GitLab E2E testing terminology. In GitLab, we no longer use the terms reliable or sanity to describe testing suites, and have since consolidated these under the term smoke

Proposal

• Rename the above test suites to the following:
  • qa-only:stg:saas-smoke
  • e2e:saas-smoke
  • e2e:production-smoke - the team may want to first discuss if e2e:production-smoke would be an appropriate name. Smoke tests typically indicate the tests are blocking, but in this case, there is nothing to block since the CustomersDot production deployment already occurred.
• Update :reliable and :production RSpec metadata tags to reflect this new naming convention (ex: :staging_smoke and :prod_smoke)
• Update documentation:
  • https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/pipeline-monitoring/
  • https://gitlab.com/gitlab-org/customers-gitlab-com/blob/main/qa/doc/pipeline_monitoring.md
  • https://gitlab.com/gitlab-org/customers-gitlab-com/blob/main/qa/doc/test_failure_debugging.md
  • https://gitlab.com/gitlab-org/customers-gitlab-com/blob/main/qa/doc/testcase_management_labels.md