As of today unable to reproduce. I asked support to check with customer. I will close issue on 2024-04-12 if no response or customer validated that issue is no longer occuring
Summary
The pipeline jobs are failing intermittently every few days with the following error in the job logs:
Running with gitlab-runner 15.8.1 (f86890c6)2 on ca2-S1 11, system ID: s_xyz3Resolving secrets00:005Preparing the "shell" executor00:006Using Shell executor...8Preparing environment00:0010ERROR: Job failed: prepare environment: exit status 0xc0000142. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
Current workaround is to periodically re-register the runner with GitLab instance, with this the runner works fine for a week or two. But this is not a long term solution.
@alejguer I bumped the severity down to 3 given this seems specific to this customer (happens on all of their nodes, in any CI configuration, but not for anyone else) and the workaround is not overly complex. But I understand how inconvenient it is. Since we don't know if it's a bug or a support issue yet given it's only showing up in their environment, I'll try to encourage getting the investigation completed so we can get a better idea on scheduling.
I've been attempting to reproduce the issue on Windows Server 2019, but unfortunately, I haven't been successful thus far. Has anyone else encountered this problem recently ?
Could you provide some additional details about your setup that would help me understand more so I can try again ?
I have a system where this is happening after some Windows and IT update. Currently running the latest GitLab Runner 17.3.1 on Windows.
In our case, we launch the gitlab-runner executable manually (it is not a service). One thing we notice is that the command prompt where the output of running gitlab-runner is gone when it happens, but the gitlab-runner process is still running in the task manager. Willing to help for troubleshooting. Can this issue be re-opened?
Dear Team,
we are facing the same issue on our Windows Server System with Gitlab runner 17.8.1 (a1fca00e).
Please reopen this ticket.
What information do you need from us?
@ratchade I have a customer ticket internal where they are able to reproduce it quite reliable. Their runners get broken and start failing jobs with this error after about a week of working without issues. I would like to reopen this issue to continue troubleshooting it. Hopefully, we could get to the bottom of it in collaboration with the customer this time.
Environment:
GitLab version v17.5.4-ee
Windows Server 2022, Version 21H2 (Build 20348.3091)
GitLab runner version: v17.8.1
executor = "shell"
shell = "pwsh"
The service is running as a user account with administrative privileges
Having the same issue since yesterday. Setup had been running like this for months without changes. About 30% of Jobs fail with this error, after a few retries, it usually works, without restart of the runner.
Setup:
Windows 11 Pro 23H2
Runner version 17.8.3 (updated yesterday morning after issue started appearing)
I've been investigating this issue to determine what might be its source.
Through the researches made on the exit status code 0xc0000142, I came across an article that explains potential causes for this status code (this article).
The article highlights that the status code is encountered when launching a process under new credentials (particularly in a Windows Service which is the case for the issue mentioned here #34376 (comment 2334668861))
It is related to an initialization failure of the application (lacks of permission, environment not properly set up for the application initialization.
This failure can system occur even when the service is running with user account with administrative privileges
The first attempt and an access denied exception right off the bat. This is the most common initial problem and is caused by the fact that the service is running under the LOCAL SYSTEM account. Strangely, the SYSTEM account is the most powerful account on the computer but one of the few things it cannot do is launch a process using CreateProcessWithLogonW which is the API underlying the call to Process.Start. So change your service account to Local Service, it's probably the more appropriate account anyway.
Is it possible to get the Application Error log related to this failure ? The command below should display the latest errors logged with this specific exit code status
On my end, I'm checking if GitLab Runner is running commands with different credentials (I could see that in some cases the failure happens during the Prepare stage)
TimeCreated Id LevelDisplayName Message----------- -- ---------------- -------2/4/2025 10:24:53 AM 25754 Information XE-Sitzung „telemetry_xevents“ wird beendet.2/4/2025 10:24:53 AM 25753 Information XE-Sitzung „telemetry_xevents“ wurde gestartet.2/4/2025 11:22:13 AM 16394 Information Offline downlevel migration succeeded.2/4/2025 11:22:43 AM 16384 Information Successfully scheduled Software Protection service for re-start at 2125-01-11T10:22:43Z. Reason: RulesEngine.2/4/2025 11:24:54 AM 25754 Information XE-Sitzung „telemetry_xevents“ wird beendet.2/4/2025 11:24:54 AM 25753 Information XE-Sitzung „telemetry_xevents“ wurde gestartet.2/4/2025 12:20:14 PM 1 Information Checking for jobs... received [0;m job[0;m=127573 repo_url[0;m=https://gitlab.eurogard.de/monitoring/code/casmessprogramm30.git runner[0;m=…2/4/2025 12:20:14 PM 1 Information Added job to processing list [0;m builds[0;m=1 job[0;m=127573 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 12:20:14 PM 2 Warning WARNING: Job failed: prepare environment: exit status 0xc0000142. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information…2/4/2025 12:20:14 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127573 job-log[0;m=0-641 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0;…2/4/2025 12:20:14 PM 1 Information Updating job... [0;m bytesize[0;m=641 checksum[0;m=crc32:b94b4eeb job[0;m=127573 runner[0;m=t3_YSNpuh…2/4/2025 12:20:15 PM 2 Warning WARNING: Submitting job to coordinator... job failed[0;m bytesize[0;m=641 checksum[0;m=crc32:b94b4eeb code[0;m=200 job[0;m=127573 job-status[0;m=failed runne…2/4/2025 12:20:15 PM 1 Information Removed job from processing list [0;m builds[0;m=0 job[0;m=127573 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 12:24:56 PM 25754 Information XE-Sitzung „telemetry_xevents“ wird beendet.2/4/2025 12:24:56 PM 25753 Information XE-Sitzung „telemetry_xevents“ wurde gestartet.2/4/2025 12:51:41 PM 16394 Information Offline downlevel migration succeeded.2/4/2025 12:52:12 PM 16384 Information Successfully scheduled Software Protection service for re-start at 2125-01-11T11:52:12Z. Reason: RulesEngine.2/4/2025 1:07:36 PM 16394 Information Offline downlevel migration succeeded.2/4/2025 1:08:15 PM 16384 Information Successfully scheduled Software Protection service for re-start at 2125-01-11T12:08:15Z. Reason: RulesEngine.2/4/2025 1:15:27 PM 9027 Information The Desktop Window Manager has registered the session port.2/4/2025 1:16:15 PM 1 Information All workers stopped. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:15 PM 2 Warning WARNING: Starting graceful shutdown, waiting for builds to finish[0;m StopSignal[0;m=quit builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:15 PM 1 Information Shutting down executor providers [0;m builds[0;m=0 max_builds[0;m=1 shutdown-timeout[0;m=30s…2/4/2025 1:16:15 PM 1 Information All executor providers shut down. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:15 PM 1 Information Can exit now! [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:28 PM 1 Information Starting multi-runner from C:\gitlab\config.toml...[0;m builds[0;m=0 max_builds[0;m=0…2/4/2025 1:16:28 PM 1 Information Configuration loaded [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:28 PM 1 Information [session_server].listen_address not defined, session endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:28 PM 1 Information Initializing executor providers [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:28 PM 1 Information listen_address not defined, metrics & debug endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 1:16:41 PM 1 Information Checking for jobs... received [0;m job[0;m=127580 repo_url[0;m=https://gitlab.eurogard.de/monitoring/code/casmessprogramm30.git runner[0;m=…2/4/2025 1:16:41 PM 1 Information Added job to processing list [0;m builds[0;m=1 job[0;m=127580 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 1:16:44 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-762 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0;…2/4/2025 1:16:47 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-831 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0;…2/4/2025 1:16:50 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-1316 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:16:54 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-1522 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:17:12 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-1959 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:17:18 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-2568 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:17:21 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-2735 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:17:24 PM 2 Warning WARNING: Job failed: exit status 1…2/4/2025 1:17:24 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127580 job-log[0;m=0-2831 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0…2/4/2025 1:17:24 PM 1 Information Updating job... [0;m bytesize[0;m=2831 checksum[0;m=crc32:d8775a87 job[0;m=127580 runner[0;m=t3_YSNpuh…2/4/2025 1:17:24 PM 2 Warning WARNING: Submitting job to coordinator... job failed[0;m bytesize[0;m=2831 checksum[0;m=crc32:d8775a87 code[0;m=200 job[0;m=127580 job-status[0;m=failed runn…2/4/2025 1:17:24 PM 1 Information Removed job from processing list [0;m builds[0;m=0 job[0;m=127580 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 1:20:08 PM 17137 Information GLPharma_DF-Datenbank wird gestartet2/4/2025 1:20:09 PM 49930 Information Parallele Wiederholung "started" für die Datenbank "GLPharma_DF" mit Workerpoolgröße [4].2/4/2025 1:20:09 PM 49930 Information Parallele Wiederholung "shutdown" für die Datenbank "GLPharma_DF" mit Workerpoolgröße [4].2/4/2025 1:20:09 PM 17573 Information CHECKDB für die GLPharma_DF-Datenbank wurde am 2012-01-23 22:03:22.723 (Ortszeit) ohne Fehler abgeschlossen. Diese Meldung dient nur zu Informationszwecken. Es ist …2/4/2025 1:20:09 PM 17137 Information Komm7128_Uni_Kiel-Datenbank wird gestartet2/4/2025 1:20:09 PM 49930 Information Parallele Wiederholung "started" für die Datenbank "Komm7128_Uni_Kiel" mit Workerpoolgröße [4].2/4/2025 1:20:09 PM 49930 Information Parallele Wiederholung "shutdown" für die Datenbank "Komm7128_Uni_Kiel" mit Workerpoolgröße [4].2/4/2025 1:25:37 PM 25754 Information XE-Sitzung „telemetry_xevents“ wird beendet.2/4/2025 1:25:37 PM 25753 Information XE-Sitzung „telemetry_xevents“ wurde gestartet.2/4/2025 2:25:38 PM 25754 Information XE-Sitzung „telemetry_xevents“ wird beendet.2/4/2025 2:25:38 PM 25753 Information XE-Sitzung „telemetry_xevents“ wurde gestartet.2/4/2025 2:51:08 PM 9027 Information The Desktop Window Manager has registered the session port.2/4/2025 2:51:15 PM 2 Warning WARNING: Starting graceful shutdown, waiting for builds to finish[0;m StopSignal[0;m=quit builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:15 PM 1 Information All workers stopped. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:15 PM 1 Information Can exit now! [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:15 PM 1 Information Shutting down executor providers [0;m builds[0;m=0 max_builds[0;m=1 shutdown-timeout[0;m=30s…2/4/2025 2:51:15 PM 1 Information All executor providers shut down. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:40 PM 1 Information Starting multi-runner from C:\gitlab\config.toml...[0;m builds[0;m=0 max_builds[0;m=0…2/4/2025 2:51:40 PM 1 Information Configuration loaded [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:40 PM 1 Information listen_address not defined, metrics & debug endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:40 PM 1 Information [session_server].listen_address not defined, session endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:40 PM 1 Information Initializing executor providers [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:51:59 PM 1 Information Checking for jobs... received [0;m job[0;m=127602 repo_url[0;m=https://gitlab.eurogard.de/monitoring/code/casmessprogramm30.git runner[0;m=…2/4/2025 2:51:59 PM 1 Information Added job to processing list [0;m builds[0;m=1 job[0;m=127602 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 2:51:59 PM 2 Warning WARNING: Job failed: prepare environment: exit status 0xc0000142. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information…2/4/2025 2:51:59 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127602 job-log[0;m=0-641 job-status[0;m=running runner[0;m=t3_YSNpuh sent-log[0;…2/4/2025 2:51:59 PM 1 Information Updating job... [0;m bytesize[0;m=641 checksum[0;m=crc32:90e23ddb job[0;m=127602 runner[0;m=t3_YSNpuh…2/4/2025 2:52:00 PM 2 Warning WARNING: Submitting job to coordinator... job failed[0;m bytesize[0;m=641 checksum[0;m=crc32:90e23ddb code[0;m=200 job[0;m=127602 job-status[0;m=failed runne…2/4/2025 2:52:00 PM 1 Information Removed job from processing list [0;m builds[0;m=0 job[0;m=127602 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 2:52:17 PM 2 Warning WARNING: Starting graceful shutdown, waiting for builds to finish[0;m StopSignal[0;m=quit builds[0;m=0 max_builds[0;m=1…2/4/2025 2:52:17 PM 1 Information All workers stopped. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:52:17 PM 1 Information All executor providers shut down. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:52:17 PM 1 Information Shutting down executor providers [0;m builds[0;m=0 max_builds[0;m=1 shutdown-timeout[0;m=30s…2/4/2025 2:52:17 PM 1 Information Can exit now! [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:53:55 PM 1 Information Starting multi-runner from C:\gitlab\config.toml...[0;m builds[0;m=0 max_builds[0;m=0…2/4/2025 2:53:55 PM 1 Information Configuration loaded [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:53:55 PM 1 Information [session_server].listen_address not defined, session endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:53:55 PM 1 Information Initializing executor providers [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:53:55 PM 1 Information listen_address not defined, metrics & debug endpoints disabled[0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:54:08 PM 1 Information Checking for jobs... received [0;m job[0;m=127603 repo_url[0;m=https://gitlab.eurogard.de/monitoring/code/casmessprogramm30.git runner[0;m=…2/4/2025 2:54:08 PM 1 Information Added job to processing list [0;m builds[0;m=1 job[0;m=127603 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 2:54:08 PM 2 Warning WARNING: Job failed: prepare environment: exit status 0xc0000142. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information…2/4/2025 2:54:08 PM 1 Information Appending trace to coordinator...ok [0;m code[0;m=202 job[0;m=127603 job-log[0;m=0-646 job-status[0;m=running runner[0;m=t3_Svvoy4 sent-log[0;…2/4/2025 2:54:08 PM 1 Information Updating job... [0;m bytesize[0;m=646 checksum[0;m=crc32:be185057 job[0;m=127603 runner[0;m=t3_Svvoy4…2/4/2025 2:54:08 PM 2 Warning WARNING: Submitting job to coordinator... job failed[0;m bytesize[0;m=646 checksum[0;m=crc32:be185057 code[0;m=200 job[0;m=127603 job-status[0;m=failed runne…2/4/2025 2:54:08 PM 1 Information Removed job from processing list [0;m builds[0;m=0 job[0;m=127603 max_builds[0;m=1 project[0;m=119 repo_url[0;m=https://gitlab.eurogard.de/m…2/4/2025 2:54:22 PM 2 Warning WARNING: Starting graceful shutdown, waiting for builds to finish[0;m StopSignal[0;m=quit builds[0;m=0 max_builds[0;m=1…2/4/2025 2:54:22 PM 1 Information All workers stopped. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:54:22 PM 1 Information Shutting down executor providers [0;m builds[0;m=0 max_builds[0;m=1 shutdown-timeout[0;m=30s…2/4/2025 2:54:22 PM 1 Information All executor providers shut down. [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:54:22 PM 1 Information Can exit now! [0;m builds[0;m=0 max_builds[0;m=1…2/4/2025 2:55:09 PM 6000 Information The winlogon notification subscriber <WSearch> was unavailable to handle a notification event.2/4/2025 2:55:12 PM 6000 Information The winlogon notification subscriber <WSearch> was unavailable to handle a notification event.2/4/2025 2:55:12 PM 6000 Information The winlogon notification subscriber <WSearch> was unavailable to handle a notification event.2/4/2025 2:55:18 PM 17147 Information SQL Server wird beendet, weil das System heruntergefahren wird. Diese Meldung dient nur zu Informationszwecken. Es ist keine Benutzeraktion erforderlich.2/4/2025 2:55:22 PM 13 Error Volume Shadow Copy Service information: The COM Server with CLSID {4e14fba2-2e22-11d1-9964-00c04fbbb345} and name CEventSystem cannot be started. [0x8007045b, Der C…
I have been unable to reproduce the issue so far despite multiple attempts. Based on user feedback, the problem can occur during the Prepare stage, and I had hoped to replicate it, but so far, I have not been successful
I am looking into other ways to reproduce the issue to get it fix.
this issue is becoming headache for us. We are not using any new process but still it is giving this error and then we are forced to restart the runner service. @ratchade