Provision and configure the gitlab-runner that will execute CI/CD jobs using docker+machine VMs, similar to the current hosted runners. ## Work Items ### Runner-manager - [x] Clean up unused secrets from [Vault](https://vault.gitlab.net/ui/vault/secrets-engines/k8s/kv/list/env/r-saas-l-p-amd64/). - [x] Delete [env/r-saas-l-p-amd64/gitlab-runner/gcs-cache](https://vault.gitlab.net/ui/vault/secrets-engines/k8s/kv/env%2Fr-saas-l-p-amd64%2Fgitlab-runner%2Fgcs-cache). - [x] Upgrade the chart to >0.88.2 :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2442 - [x] Convert configs to use `configOverride` after https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/544+ :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2481+ - [x] Delete the `runner-registration-token` from [env/r-saas-l-p-amd64/gitlab-runner/runner](https://vault.gitlab.net/ui/vault/secrets-engines/k8s/kv/env%2Fr-saas-l-p-amd64%2Fgitlab-runner%2Frunner/details?version=2) - [x] Re-add sentry secrets after [sentry DSN env support](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6899) :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2617+ - [x] Deploy `gitlab-runner` via ArgoCD - [x] GCP Credentials and network access workload cluster - [x] ~~Kubernetes~~ docker+machine executor - [x] Sufficient job timeouts and termination grace periods :point_right: https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/546 and https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/1780 - [x] Tune concurrency to a low value so that we can limit how much of the job volume we get - [x] Validate zero-downtime deployments - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2609+ - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2610+ - [x] Enable [docker-machine drain](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6330) :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2616+ - [ ] Observability: - [x] Single vs Multi-tenant decision + add recording rules - [x] https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/14672 - [x] https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/14685 - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2557+ - [x] https://gitlab.com/gitlab-com/runbooks/-/merge_requests/10985+ - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2612+ - [x] Send logs to Kibana - [x] Add new dedicated SA :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/14631 - [x] Enable logging on the runner-manager :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2534+ - [x] Fix the missing labels :point_right: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/2556+ - [ ] wiz sensor support - [x] Register group-level runner managers ### Workloads - [ ] Create a k8s cluster in the QA project - [ ] Create a shard for QA runners in ArgoCD - [ ] Gradual workload migration