Runner Test Gaps
# Test Gap Analysis for Runner
Team - we're moving this to a Google Sheet to make it easier to contribute. Sheet can be accessed here:
https://docs.google.com/spreadsheets/d/1Hy__yHmdyMcZAES0vdqCaErE3xZHLONiJSnvfLEOvCM/edit#gid=1968451650
## Known Test Coverage and Gaps
| Facet | Enterprise-intended | Unit Testing | Component Integration | Integration (binary execution) | E2E/Staging | Test Coverage satisfaction | Incidents? |
|-----------------------------------------------------------------------------------------------------------------|---------------------|--------------------|-----------------------|-----------------------------------------|-------------|---------------------------------------------------------------|------------|
| environment | Dev-level (pre-release) runner builds on .com environments for E2E | | | | | :x: | | |
| environment | air-gapped | | :white_check_mark: | | | :x: | N | |
| Kubernetes executor | :white_check_mark: | :white_check_mark: | :white_check_mark: | runner-incept | NA | N | |
| Docker-machine executor | :white_check_mark: | :white_check_mark: | | :warning: | NA | N | |
| Docker executor | :white_check_mark: | :x: | | runner-incept | NR | N | |
| Custom executor Shell | :white_check_mark: | :white_check_mark: | | runner-incept | NR | N | |
| Docker-ssh+machine executor | :white_check_mark: | | | runner-incept | NR | N | |
| Docker-ssh executor | :white_check_mark: | | | runner-incept | NR | N | |
| SSH executor | :white_check_mark: | | | runner-incept | NR | N | |
| AWS Fargate Driver | :white_check_mark: | | | :x: | NA | N | |
| Autoscaling SaaS | :white_check_mark: | | | :x: | :x: | N | |
| Helper images | :white_check_mark: | :white_check_mark: | | runner-incept | NA | N | |
| Vault | :white_check_mark: | :white_check_mark: | | :x: | NA | N | |
| Helm Chart | :white_check_mark: | | | :x: | NA | N | |
| Registration | :white_check_mark: | :x: | | runner-incept | NR | N | |
| Shell forward/backward slash handling | :white_check_mark: | | | runner-incept | NR | N | |
| Windows PowerShell Core | :white_check_mark: | :x: | | runner-incept | NR | N | |
| GitLab runner service installation | :white_check_mark: | :x: | | runner-incept | NA | N | |
| Docker images running gitlab-runner | :white_check_mark: | :x: | | runner-incept | NR | N | |
| cache | :white_check_mark: | :white_check_mark: | | gitlab-org/ci-cd/tests/runner-incept#9 | NA | N | |
| arm64 | :white_check_mark: | | :x: | runner-incept | NA | N | |
| s390x | :white_check_mark: | | :x: | runner-incept | NA | N | |
| Docker image executor compatibility matrix | :white_check_mark: | | :x: | runner-incept | NA | N | |
| Backwards compatibility matrix | :white_check_mark: | | :x: | | :x: | N | |
| Macstadium integration | :white_check_mark: | | | | | N | |
| Parallels executor | | | | :x: | NA | N | |
| Container daemon compatibility | :white_check_mark: | | | :x: | NA | N | |
| VirtualBox executor | | | | | NA | N | |
| Container entrypoint overrides | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#17 | NA | N | |
| Package validation | :white_check_mark: | | | runner-incept | NA | N | |
| RedHat OpenShift operator | :white_check_mark: | | :x: | :x: | NA | N | |
| FreeBSD | | | | gitlab-org/ci-cd/tests/runner-incept#1 | NA | N | |
| macOS | | | :x: | :x: | NA | N | |
| Windows compatibility | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#3 | NR | N - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4467 | |
| Autoscaling - Cloud compatibility | :white_check_mark: | | | :x: | NA | N | |
| Registration compat matrix - Driver/Feature | :white_check_mark: | | | runner-incept | NA | N | |
| Command validation | :white_check_mark: | | :x: | runner-incept | NA | N | |
| Runner feature flags | :white_check_mark: | | :x: | runner-incept | NR | N | |
| Monitoring - Embedded Prometheus | :white_check_mark: | :white_check_mark: | :x: | runner-incept | NA | N | |
| Monitoring - Embedded pprof | :white_check_mark: | :white_check_mark: | :x: | runner-incept | NA | N | |
| Shell compat matrix | :white_check_mark: | | | runner-incept | NA | N | |
| Component compat matrix - git | :white_check_mark: | | | runner-incept | NA | N | |
| Component compat matrix - kubernetes API | :white_check_mark: | | | runner-incept | NA | N | |
| Log limits | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#14 | NA | N | |
| Log handling - variables | :white_check_mark: | :white_check_mark: | | runner-incept | NR | N | |
| Variable handling - inheritance | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#15 | NR | N | |
| Variable handling - file type | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#16 | NR | N | |
| API - HTTP response handling | :white_check_mark: | | | runner-incept | NR | N | |
| Services compat matrix | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#8 | NR | N | |
| Secrets management | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#14 | NA | N | |
| Artifact handling | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#10 | NA | N | |
| Timeout keyword | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#11 | NR | N | |
| Image keyword | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#7 | NR | N | |
| K8s non-root images | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#12 | NA | N | |
| Allow_failure/exit_codes | :white_check_mark: | | | gitlab-org/ci-cd/tests/runner-incept#6 | NR | N | |
| [multi pull policy (docker)](https://docs.gitlab.com/runner/executors/docker.html#using-multiple-pull-policies) | :white_check_mark: | :white_check_mark: | :x: | | NR | N | |
| [multi pull policy (Kubernetes)](https://docs.gitlab.com/runner/executors/kubernetes.html#using-pull-policies) | :white_check_mark: | :white_check_mark: | :x: | | NR | N | |
### Key
| Code | Meaning |
| :-----: | :----- |
| :white_check_mark: | Satisfactory coverage |
| :x: | Testing required at this level but coverage insufficient |
| NA | Not applicable or appropriate at this test level |
| NR | Can be tested at this level, but lower level coverage should be sufficient |
| runner-incept | Can be satisfactorily covered in `runner-incept` |
| | Requires investigation |
## To Be Done
- [ ] Additional research on unknown coverage levels
- [ ] Areas with coverage need links to specs
- [ ] Items tagged `runner incept` need corresponding issues created
- [ ] Incidents from analysis need to be mapped to this list (they are in there, just haven't added those links yet)
- [ ] Identify highest risk areas for risk mapping document
epic