[macos] GitLab runner fails to honnor git credentials store when run as Launch Daemons

Summary

When running the gitlab runner as a Launch Deamon, it fails to respect the git config [credential] helper=store and tries to get git credentials from a macOS keychain. This blocks the runner on the git fetch operation.

Here is my gitconfig configured for store

% git config credential.helper
store

but despite these, the runner blocks and launches a process that tries to use osxkeychain

% ps ax | grep osxkeychain
41755   ??  S      0:00.01 git credential-osxkeychain store
41756   ??  S      0:00.01 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-credential-osxkeychain store

I know Launch Daemon are officially not supported, but until recently, it worked.

This behavior happens only when launching the runner as a daemon. When launching the same runner, with the same config from an SSH session (no GUI, no Terminal app involved), it works.

Steps to reproduce

.gitlab-ci.yml

stages:
  - build
  - test
  - post-build

variables:
  GIT_CLEAN_FLAGS: none # do not git clean to preserve files between stage

build:       
  stage: build
  script:
    - ./ci_actions/03_build.sh

local-test: 
  stage: test    
  script:
    - ./ci_actions/04_local_tests.sh

cleanup:
    stage: post-build
    script:
        - ./ci_actions/07_cleanup.sh

daemon.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>gitlab.runner.workshop</string>
    <key>KeepAlive</key>
    <dict>
      <key>SuccessfulExit</key>
      <false/>
    </dict>    
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/bin/gitlab-runner</string>
      <string>run</string>
      <string>--working-directory</string>
      <string>/Users/ec2-user/gitlab-runner</string>
      <string>--config</string>
      <string>/Users/ec2-user/.gitlab-runner/config.toml</string>
      <string>--service</string>
      <string>gitlab-runner</string>
      <!-- <string>--syslog</string> -->
    </array>
    <key>UserName</key>
    <string>ec2-user</string>
    <key>WorkingDirectory</key>
    <string>/Users/ec2-user/gitlab-runner</string>
    <key>RunAtLoad</key>
    <true/>    
    <key>StandardOutPath</key>
    <string>/Users/ec2-user/gitlab-runner/out.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/ec2-user/gitlab-runner/err.log</string>
    <key>EnvironmentVariables</key>
    <dict>
      <key>PATH</key>
      <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
    </dict>      
    <!-- this is required to access macOS keychain for code signature -->
    <key>SessionCreate</key>
    <true/>
  </dict>
</plist>

Actual behavior

The runner is blocked. In the process list, we see a git credentials-osxkeychain store. I suspect this process is blocking the runner because it expects inputs from stdin

41716   ??  Ss     0:00.01 /bin/bash /Users/ec2-user/gitlab-runner.sh
41719   ??  S      0:00.10 gitlab-runner run --working-directory /Users/ec2-user/gitlab-runner --config /Users/ec2-user/.gitlab-runner/config.toml
41752   ??  S      0:00.01 /Applications/Xcode.app/Contents/Developer/usr/bin/git -c http.userAgent=gitlab-runner 16.4.1 darwin/arm64 fetch origin +refs/pipelines/19:refs/pipelines/19 +refs/heads/main:refs/remotes/origin/main --depth 20 --prune --quiet
41753   ??  S      0:00.00 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git remote-https origin https://gitlab-ci-token:64_oDBxEk_R69eSzy65xoSq@gitlab-server-gitlab-7cecc7882511fab4.elb.us-east-2.amazonaws.com/awsworkshop/iOS_SampleApp.git
41754   ??  S      0:00.01 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-remote-https origin https://gitlab-ci-token:64_oDBxEk_R69eSzy65xoSq@gitlab-server-gitlab-7cecc7882511fab4.elb.us-east-2.amazonaws.com/awsworkshop/iOS_SampleApp.git
41755   ??  S      0:00.01 git credential-osxkeychain store
41756   ??  S      0:00.01 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-credential-osxkeychain store

Expected behavior

I expect the gitlab runner to respect the configuration in ~/.gitconfig and to not block while waiting for an input.

cat ~/.gitconfig 
[credential]
	helper = store

 % git config credential.helper
store

Environment description

This is a self hosted runner on Amazon EC2 Mac. These are build machine, there is no GUI session associated, only SSH-like access through Amazon SSM.

git is provided by Xcode (I tested version 14.3.1 and 15.0).

When starting the runner from a command line SSM session, it works as expected.
When starting it as a launch daemon (see plist file above), it fails and blocks on the git clone operation.

config.toml contents

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab.runner.workshop"
  url = "https://redacted.elb.us-east-2.amazonaws.com"
  id = 4
  token = "glrt-redacted"
  token_obtained_at = 2023-10-17T20:37:12Z
  token_expires_at = 0001-01-01T00:00:00Z
  tls-ca-file = "/etc/gitlab-runner/certs/redacted.elb.us-east-2.amazonaws.com.crt"
  executor = "shell"
  [runners.cache]
    MaxUploadedArchiveSize = 0

Used GitLab Runner version

Version:      16.4.1
Git revision: d89a789a
Git branch:   16-4-stable
GO version:   go1.20.5
Built:        2023-10-06T01:26:32+0000
OS/Arch:      darwin/arm64

Possible fixes

n/a