Runner staging environment
Proposal
Setup a staging environment to enable runner-specific testing. This can be used for:
- deployment testing
- integration testing
- E2E testing
Requirements
- Similar to production, a customer-like environment
- Based on reference architectures
- Not used for load testing, so smallest reference architecture should be feasible
- Private to runner team
- Important for deployment testing
- Starting point for simplicity, expand controls to allow for other testing as necessary
- Configurable
- Rake/db migrations to avoid unnecessary UI setup (considering)
- Change any system component as necessary
- Runner (required at start)
- GitLab (required at start)
- Gitlay
- git
- kubernetes
- container runtimes
- GET will work for initial setup, investigation required for ongoing configurability
- Monitored
- Standard monitoring (Prometheus, et.al.)
- Testability (future)
- Clear versioning reported in tests, etc.
- Blocking tests reported automatically to Slack channel specific to runner devs
- Automated via runner pipeline
- Investigate running constantly vs spin up/down based on test needs
- Start with constant
- Testing required of GET reliability
- Investigate running constantly vs spin up/down based on test needs
- Tests
- Deployment-blocking
- Smoke tests (integration/E2E)
- Reliable tests (integration/E2E)
- Gate for merging to runner master (main)
- Integration (runner incept, etc)
- Non-blocking
- Integration
- E2E
- Deployment-blocking
Action items
-
Select GCP env and determine credentials for GET -
Create new 1k reference architecture using GET -
Setup basic monitoring (Prometheus, etc) -
Investigate configurability via GET - leverage QE Performance team
- determine workaround for current use if necessary
-
Move CI-CD Runner project tests to this environment -
Trigger based on runner repo activity -
Primary: Merge to master/main -
Secondary: Parameterize for MRs
-
-
-
Standardize on testing framework -
Make accessible/adoptable to runner development team
-
-
Develop smoke/reliable tests for blocking deployments -
Develop reliable tests for blocking deployment to master -
Develop additional non-blocking tests to run against MRs