## Summary During a recent deployment, EOC was alerted with the following Alert: >Firing 1 - Containers for the `git` service, `main` are unable to unable to start. More than 50% of the deployment's maxSurge setting consists of containers unable to start for reasons other than ContainerCreating. Prometheus showed the following, [query](https://prometheus-gke.gprd-us-east1-b.gitlab.net/graph?g0.expr=sum%20by(type%2C%20env%2C%20tier%2C%20stage%2C%20cluster)%20(kube_pod_container_status_waiting_reason%3Alabeled%7Breason!%3D%22ContainerCreating%22%2Cstage!%3D%22%22%2Ctier!%3D%22%22%2Ctype!%3D%22%22%7D)%20%3E%200%20%3E%3D%20on(type%2C%20env%2C%20tier%2C%20stage%2C%20cluster)%20(topk%20by(type%2C%20env%2C%20tier%2C%20stage%2C%20cluster)%20(1%2C%20kube_deployment_spec_strategy_rollingupdate_max_surge%3Alabeled%7Bstage!%3D%22%22%2Ctier!%3D%22%22%2Ctype!%3D%22%22%7D)%20*%200.5)&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=2h):  A quick investigation showed the following errors in the affected pods (more context here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6268#note_829420838) >`Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0bee02cbc1e95d8ed1095632c0948dd8ee0eaae233793754fde926965a218a0c": failed to find plugin "calico" in path [/home/kubernetes/bin]` This issue's aim is to understand this error and decide if there's any actions that should be taken. <!-- Give context for what problem this issue is trying to prevent from happening again. Provide a brief assessment of the risk (chance and impact) of the problem that this corrective action fixes, to assist with triage and prioritization. --> ## Related Incident(s) <!-- Note the originating incident(s) and link known related incidents/other issues --> Originating issue(s): production#6268 ## Desired Outcome/Acceptance criteria <!-- How will you know that this issue is complete? If you have any initial thoughts on implementation details e.g. what to do or not do, gotchas, edge cases etc, please share them while they are fresh in your mind. --> - [ ] The error observed is understood. - [ ] Evaluate whether there's any action that can be taken to prevent this from re-occurring. ## Associated Services <!-- Apply the appropriate services associate with this corrective action if appliable. ~Service::SERVICE_NAME --> * production-engineering~13297914 ## Corrective Action Issue Checklist * [x] link the incident(s) this corrective action arose out of * [x] give context for what problem this corrective action is trying to prevent from re-occurring * [x] assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') * [x] assign a priority (this will default to 'priority::4')