## Summary As described by @kakawait in the [forum thread here](https://forum.gitlab.com/t/high-number-of-error-job-failed-system-failure-context-deadline-exceeded-since-19-1-0/134287), after updating to GitLab runner 19.1.0, we're experiencing CI failures: ``` ERROR: Job failed (system failure): context deadline exceeded ``` This is a follow-up to [this discussion](https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39426#note_3478872728) ## Relevant logs and/or screenshots - The duration of the entire job was 2 minutes 15 seconds, I'm not aware of any explicit timeouts configured (`prepare_timeout`, `get_sources_timeout`, `RUNNER_SCRIPT_TIMEOUT`, etc.) or feature flags enabled. - Please note that the failed job doesn't run the `Executing "step_script" stage of the job script`: {width=530 height=600} <details> <summary>job log of a failed job (slightly redacted)</summary> ```sh Running with gitlab-runner 19.1.0 (5eb085ab) on imx8mp-iot-gate11 hhw7hniIT, system ID: s_d5040340a0e9 Resolving secrets Preparing the "docker" executor 00:01 Using Docker executor with image registry.gitlab.com/gapfruit/development/genode/sdk/sculpt:pipeline-608 ... Using effective pull policy of [if-not-present] for container registry.gitlab.com/gapfruit/development/genode/sdk/sculpt:pipeline-608 Using locally found image version due to "if-not-present" pull policy Using docker image sha256:cfd653a652a66de6d851b58a8c8087b38fb72612e73cc1066d4f3d75d88d8f24 for registry.gitlab.com/gapfruit/development/genode/sdk/sculpt:pipeline-608 with digest registry.gitlab.com/gapfruit/development/genode/sdk/sculpt@sha256:717656693b1909fe38253128d2d8ce88f7bf20b6bc6d95ebfeba55eb96579d12 ... Preparing environment 00:01 Using effective pull policy of [if-not-present] for container sha256:bb91836945eb72c3c663b859f360f6a297578e34db77a5805918f813da6b2299 Running on runner-hhw7hniit-project-5419404-concurrent-0 via ci-runner-host1... Getting source from Git repository 00:07 Gitaly correlation ID: a0faff4b1e1bfcc9-ZRH Fetching changes... Initialized empty Git repository in /builds/gapfruit/development/gapfruit/.git/ Created fresh repository. Checking out 3dc4a567 as detached HEAD (ref is staging)... Skipping Git submodules setup Restoring cache 00:02 Checking cache for imx8mp_iot_gate.gapfruit.rite-199-protected... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. WARNING: Cache file does not exist Failed to extract cache Downloading artifacts 01:40 Downloading artifacts for genode.sdk (14965921719)... Direct download from cdn.artifacts.gitlab-static.net correlation_id=a0faff98efb1b571-ZRH id=14965921719 status=302 token=70_suBguy Downloading artifacts from coordinator... ok correlation_id=a0faff98efb1b571-ZRH id=14965921719 status=302 token=70_suBguy Downloading artifacts for depot (14965921727)... Direct download from cdn.artifacts.gitlab-static.net correlation_id=a0faffa92d2aa285-ZRH id=14965921727 status=302 token=70_suBguy Downloading artifacts from coordinator... ok correlation_id=a0faffa92d2aa285-ZRH id=14965921727 status=302 token=70_suBguy Downloading artifacts for arm_v8a.depot (14965921729)... Direct download from cdn.artifacts.gitlab-static.net correlation_id=a0fb0135fe44bb10-ZRH id=14965921729 status=302 token=70_suBguy Downloading artifacts from coordinator... ok correlation_id=a0fb0135fe44bb10-ZRH id=14965921729 status=302 token=70_suBguy Saving cache for failed job 00:11 Creating cache imx8mp_iot_gate.gapfruit.rite-199-protected... WARNING: ccache: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gapfruit/development/gapfruit) WARNING: No files to cache. Created cache Uploading artifacts for failed job 00:04 Uploading artifacts... WARNING: ...: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gapfruit/development/gapfruit) ... ERROR: No files to upload Cleaning up project directory and file based variables 00:07 ERROR: Job failed (system failure): context deadline exceeded ``` </details> ## Environment description <details> <summary> output of lsb_release on the runner host system </summary> ``` Distributor ID: Debian Description: Debian GNU/Linux 13 (trixie) Release: 13 Codename: trixie ``` </details> <!-- Please add the contents of `config.toml` inside of the code blocks (```) below, remember to remove any secret tokens! --> <details> <summary> config.toml contents </summary> ```toml concurrent = 6 check_interval = 0 connection_max_age = "15m0s" shutdown_timeout = 0 [session_server] session_timeout = 1800 ... other runners ... [[runners]] name = "imx8mp-iot-gate11" limit = 1 url = "https://gitlab.com" id = "REDACTED" token = "REDACTED" token_obtained_at = "REDACTED" token_expires_at = "REDACTED" executor = "docker" environment = ["BOARD_NAME=imx8mp-iot-gate11"] [runners.cache] MaxUploadedArchiveSize = 0 [runners.cache.s3] AssumeRoleMaxConcurrency = 0 [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "debian:13" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] volume_keep = false pull_policy = ["if-not-present"] shm_size = 0 network_mtu = 0 ``` </details> ### Used GitLab Runner version ``` Version: 19.1.0 Git revision: 5eb085ab Git branch: 19-1-stable GO version: go1.26.3 Built: 2026-06-16T21:55:37Z OS/Arch: linux/amd64 ```