## Release notes {placeholder for release notes} ### Overview In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2351 there were networking problems between GitLab and DockerHub which resulted in the `docker pull $IMAGE` command to take an **extremely** long time since there were no timeouts set in place by the Runner. The docker pull ended up taking the duration of the job, so if the job timeout was 3 hours, the job took 3 hours until it finished trying to pull the image. This caused a cascading effect because the available number of concurrent jobs was filled up by these jobs pulling forever. ## Proposal - Allow user to define a configurable timeout for the whole `prepare` stage of a single job execution. Note - by introducing the timeout for the prepare stage, as noted [here](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26583#note_504305689), this would be a generic solution that will be used by all CI jobs regardless of the executor that is used. - Introduce a default limit of 15 minutes. The assumption is that 15 minutes is enough time to prepare the environment for docker+linux. Docker+windows may need a longer timeout. - Enable the default timeout to be changed in the `[[runners]]` level in the runner's `config.toml` file. in case of the Docker executor, the `prepare` stage would be limited mostly by the time of pulling the images and spinning-up the service containers ### Expected behavior when implemented - **Docker executor**: in case of the Docker executor, the `prepare` stage would be limited mostly by the time of pulling the images and spinning-up the service containers. - **Docker machine executo**r: in case of the Docker Machine executor it would additionally include also the time needed to spin-up the autoscaled VM if it's the case with the provided autoscaling configuration. - **VirtualBox/Parallels**: in case of the VirtualBox/Parallels executors it would mostly relay on the time needed to spin-up a new VM. - **SSH & Shell**: in case of the SSH and Shell executors the `prepare` stage should be mostly innoticeable. - **Custom executor**: in case of the Custom executor, it fully depends on the driver's configuration ## Original proposal (included here to provide additional context) - Implement a timeout when pulling a docker image for the ~"executor::docker" of around 15-30 minutes so that the job fails if it takes longer than that time (if it has not pulled the image by then it's safe to say the image isn't going to be pulled). This would result in the Runner being more resilient to packet loss between the Runner and where it's pulling the image. If a timeout is implemented the job finishes at a failed state much quicker and we don't end up putting back pressure on the job queues. - We need to think carefully about the timeout here and make sure that we have enough time to pull Windows images that are sometimes 5GB large. If the `docker pull` logic has a way to see if we are getting any data from the network request or not it might be ideal to see if we can have some kind of checks to see when the last byte was written. - We should also investigate how Kubernetes does this internally and try to mimic this since Kubernetes is resilient to such problems and just shows an error to the user when it can't pull an image.