Runner Test Gaps

Test Gap Analysis for Runner

Team - we're moving this to a Google Sheet to make it easier to contribute. Sheet can be accessed here:

https://docs.google.com/spreadsheets/d/1Hy__yHmdyMcZAES0vdqCaErE3xZHLONiJSnvfLEOvCM/edit#gid=1968451650

Known Test Coverage and Gaps

Facet	Enterprise-intended	Unit Testing	Component Integration	Integration (binary execution)	E2E/Staging	Test Coverage satisfaction	Incidents?
environment	Dev-level (pre-release) runner builds on .com environments for E2E					❌
environment	air-gapped		✅			❌	N
Kubernetes executor	✅	✅	✅	runner-incept	NA	N
Docker-machine executor	✅	✅		⚠	NA	N
Docker executor	✅	❌		runner-incept	NR	N
Custom executor Shell	✅	✅		runner-incept	NR	N
Docker-ssh+machine executor	✅			runner-incept	NR	N
Docker-ssh executor	✅			runner-incept	NR	N
SSH executor	✅			runner-incept	NR	N
AWS Fargate Driver	✅			❌	NA	N
Autoscaling SaaS	✅			❌	❌	N
Helper images	✅	✅		runner-incept	NA	N
Vault	✅	✅		❌	NA	N
Helm Chart	✅			❌	NA	N
Registration	✅	❌		runner-incept	NR	N
Shell forward/backward slash handling	✅			runner-incept	NR	N
Windows PowerShell Core	✅	❌		runner-incept	NR	N
GitLab runner service installation	✅	❌		runner-incept	NA	N
Docker images running gitlab-runner	✅	❌		runner-incept	NR	N
cache	✅	✅		gitlab-org/ci-cd/tests/runner-incept#9	NA	N
arm64	✅		❌	runner-incept	NA	N
s390x	✅		❌	runner-incept	NA	N
Docker image executor compatibility matrix	✅		❌	runner-incept	NA	N
Backwards compatibility matrix	✅		❌		❌	N
Macstadium integration	✅					N
Parallels executor				❌	NA	N
Container daemon compatibility	✅			❌	NA	N
VirtualBox executor					NA	N
Container entrypoint overrides	✅			gitlab-org/ci-cd/tests/runner-incept#17	NA	N
Package validation	✅			runner-incept	NA	N
RedHat OpenShift operator	✅		❌	❌	NA	N
FreeBSD				gitlab-org/ci-cd/tests/runner-incept#1	NA	N
macOS			❌	❌	NA	N
Windows compatibility	✅			gitlab-org/ci-cd/tests/runner-incept#3	NR	N - #4467 (closed)
Autoscaling - Cloud compatibility	✅			❌	NA	N
Registration compat matrix - Driver/Feature	✅			runner-incept	NA	N
Command validation	✅		❌	runner-incept	NA	N
Runner feature flags	✅		❌	runner-incept	NR	N
Monitoring - Embedded Prometheus	✅	✅	❌	runner-incept	NA	N
Monitoring - Embedded pprof	✅	✅	❌	runner-incept	NA	N
Shell compat matrix	✅			runner-incept	NA	N
Component compat matrix - git	✅			runner-incept	NA	N
Component compat matrix - kubernetes API	✅			runner-incept	NA	N
Log limits	✅			gitlab-org/ci-cd/tests/runner-incept#14	NA	N
Log handling - variables	✅	✅		runner-incept	NR	N
Variable handling - inheritance	✅			gitlab-org/ci-cd/tests/runner-incept#15	NR	N
Variable handling - file type	✅			gitlab-org/ci-cd/tests/runner-incept#16	NR	N
API - HTTP response handling	✅			runner-incept	NR	N
Services compat matrix	✅			gitlab-org/ci-cd/tests/runner-incept#8	NR	N
Secrets management	✅			gitlab-org/ci-cd/tests/runner-incept#14	NA	N
Artifact handling	✅			gitlab-org/ci-cd/tests/runner-incept#10	NA	N
Timeout keyword	✅			gitlab-org/ci-cd/tests/runner-incept#11	NR	N
Image keyword	✅			gitlab-org/ci-cd/tests/runner-incept#7	NR	N
K8s non-root images	✅			gitlab-org/ci-cd/tests/runner-incept#12	NA	N
Allow_failure/exit_codes	✅			gitlab-org/ci-cd/tests/runner-incept#6	NR	N
multi pull policy (docker)	✅	✅	❌		NR	N
multi pull policy (Kubernetes)	✅	✅	❌		NR	N

Key

Code	Meaning
✅	Satisfactory coverage
❌	Testing required at this level but coverage insufficient
NA	Not applicable or appropriate at this test level
NR	Can be tested at this level, but lower level coverage should be sufficient
runner-incept	Can be satisfactorily covered in runner-incept
	Requires investigation

To Be Done

• Additional research on unknown coverage levels
• Areas with coverage need links to specs
• Items tagged runner incept need corresponding issues created
• Incidents from analysis need to be mapped to this list (they are in there, just haven't added those links yet)
• Identify highest risk areas for risk mapping document