Runner staging environment

Proposal

Setup a staging environment to enable runner-specific testing. This can be used for:

  • deployment testing
  • integration testing
  • E2E testing

Requirements

  • Similar to production, a customer-like environment
    • Based on reference architectures
    • Not used for load testing, so smallest reference architecture should be feasible
  • Private to runner team
    • Important for deployment testing
    • Starting point for simplicity, expand controls to allow for other testing as necessary
  • Configurable
    • Rake/db migrations to avoid unnecessary UI setup (considering)
    • Change any system component as necessary
      • Runner (required at start)
      • GitLab (required at start)
      • Gitlay
      • git
      • kubernetes
      • container runtimes
    • GET will work for initial setup, investigation required for ongoing configurability
  • Monitored
    • Standard monitoring (Prometheus, et.al.)
    • Testability (future)
      • Clear versioning reported in tests, etc.
    • Blocking tests reported automatically to Slack channel specific to runner devs
  • Automated via runner pipeline
    • Investigate running constantly vs spin up/down based on test needs
      • Start with constant
      • Testing required of GET reliability
  • Tests
    • Deployment-blocking
      • Smoke tests (integration/E2E)
      • Reliable tests (integration/E2E)
    • Gate for merging to runner master (main)
      • Integration (runner incept, etc)
    • Non-blocking
      • Integration
      • E2E

Action items

  • Select GCP env and determine credentials for GET
  • Create new 1k reference architecture using GET
  • Setup basic monitoring (Prometheus, etc)
  • Investigate configurability via GET
    • leverage QE Performance team
    • determine workaround for current use if necessary
  • Move CI-CD Runner project tests to this environment
    • Trigger based on runner repo activity
      • Primary: Merge to master/main
      • Secondary: Parameterize for MRs
  • Standardize on testing framework
    • Make accessible/adoptable to runner development team
  • Develop smoke/reliable tests for blocking deployments
  • Develop reliable tests for blocking deployment to master
  • Develop additional non-blocking tests to run against MRs