Runner staging environment

Proposal

Setup a staging environment to enable runner-specific testing. This can be used for:

• deployment testing
• integration testing
• E2E testing

Requirements

• Similar to production, a customer-like environment
  • Based on reference architectures
  • Not used for load testing, so smallest reference architecture should be feasible
• Private to runner team
  • Important for deployment testing
  • Starting point for simplicity, expand controls to allow for other testing as necessary
• Configurable
  • Rake/db migrations to avoid unnecessary UI setup (considering)
  • Change any system component as necessary
    • Runner (required at start)
    • GitLab (required at start)
    • Gitlay
    • git
    • kubernetes
    • container runtimes
  • GET will work for initial setup, investigation required for ongoing configurability
• Monitored
  • Standard monitoring (Prometheus, et.al.)
  • Testability (future)
    • Clear versioning reported in tests, etc.
  • Blocking tests reported automatically to Slack channel specific to runner devs
• Automated via runner pipeline
  • Investigate running constantly vs spin up/down based on test needs
    • Start with constant
    • Testing required of GET reliability
• Tests
  • Deployment-blocking
    • Smoke tests (integration/E2E)
    • Reliable tests (integration/E2E)
  • Gate for merging to runner master (main)
    • Integration (runner incept, etc)
  • Non-blocking
    • Integration
    • E2E

Action items

• Select GCP env and determine credentials for GET
• Create new 1k reference architecture using GET
• Setup basic monitoring (Prometheus, etc)
• Investigate configurability via GET
  • leverage QE Performance team
  • determine workaround for current use if necessary
• Move CI-CD Runner project tests to this environment
  • Trigger based on runner repo activity
    • Primary: Merge to master/main
    • Secondary: Parameterize for MRs
• Standardize on testing framework
  • Make accessible/adoptable to runner development team
• Develop smoke/reliable tests for blocking deployments
• Develop reliable tests for blocking deployment to master
• Develop additional non-blocking tests to run against MRs