MATLAB process preventing GitLab Runner job completion with Shell executor

Summary

When using the Shell executor in GitLab Runner, jobs that execute MATLAB commands hang indefinitely if MATLAB's background processes are initially started by the runner. The job only completes successfully if MATLAB's background processes were first initialized through a terminal session.

When MATLAB processes are first initialized by the runner:

  1. The MATLAB command executes successfully and takes about 1 minute to complete

  2. Three processes are spawned:

gitlab-+  463088  0.4  0.4 3168004 140372 ?      Sl   20:18   0:20 /home/gitlab-runner/.MathWorks/ServiceHost/-mw_shared_installs/v2024.8.0.2/bin/glnxa64/MathWorksServiceHost service --realm-id companion@prod@production --detached
gitlab-+  463222  0.1  0.1 934296 52320 ?        Sl   20:18   0:04 /home/gitlab-runner/.MathWorks/ServiceHost/-mw_shared_installs/v2024.8.0.2/bin/glnxa64/MathWorksServiceHost-Monitor --client-id 463088 --realm-id companion@prod@production --lifetime-token 37 -1
gitlab-+  530202  118  0.2 1175596 68640 ?       Sl   21:31   0:00 /home/gitlab-runner/.MathWorks/ServiceHost/-mw_shared_installs/v2024.8.0.2/bin/glnxa64/MathWorksServiceHost client-v1 --lifetime-token 4 -1 --realm-id companion@prod@production --locale en_US --application-id Mathworks.MATLAB.MATLAB.R2024a --client-id 530057 --matlabrootpath /usr/local/MATLAB/R2024a --nodisplay --gui tty

  1. The client process (PID 530202) exits after command completion

  2. Two background processes continue running:

  • MathWorksServiceHost service
  • MathWorksServiceHost-Monitor
  1. The job hangs indefinitely and never completes

Steps to reproduce

  1. Configure a GitLab Runner with Shell executor
  2. Ensure MATLAB R2024a is installed and accessible to the gitlab-runner user
  3. Create a job that executes a MATLAB command
  4. Run the job without having previously initialized MATLAB from terminal

Sample MATLAB file: decode_analyze_field_data.m

% decode_analyze_field_data.m

% Generate some sample field data
field_data = randn(100, 1);  % 100 random data points with normal distribution

% Perform analysis
mean_value = mean(field_data);
std_value = std(field_data);

% Display results
fprintf('Mean of field data: %.2f\n', mean_value);
fprintf('Standard deviation of field data: %.2f\n', std_value);

% Save results to a file
save('analysis_results.mat', 'mean_value', 'std_value');
.gitlab-ci.yml 
run:
  stage: test
  script:
    - matlab -batch "decode_analyze_field_data"
  tags:
    - matlab

Actual behavior

Job hangs indefinitely

Expected behavior

The job should complete successfully after the MATLAB command finishes executing, regardless of whether MATLAB's background processes were initialized by the runner or through terminal.

Relevant logs and/or screenshots

job log 
Add the job log

Environment description

  • MATLAB Version: R2024a
  • GitLab Runner Executor: Shell
config.toml contents 
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "<runner name>"
  url = "<gitlab-url>"
  id = 68
  token = "glrt-xxxx..."
  token_obtained_at = 2024-06-14T13:55:20Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Used GitLab Runner version

Current workaround

Initialize MATLAB by running a command as the gitlab-runner user in terminal before executing the job:

  1. Switch to gitlab-runner user
  2. Execute any MATLAB command from terminal
  3. Leave the background processes running
  4. Run the GitLab job

This allows subsequent runner jobs to complete successfully, even though the background processes remain running.

Possible fixes

  1. If the background processes are killed and the runner is allowed to restart them, the job hanging behavior reproduces
  2. The job only completes successfully when the background processes are started via terminal, not when started by the runner
  3. The actual MATLAB command execution is successful in both cases, the issue is only with job completion
  4. The behavior is consistent across multiple test runs

The Shell executor may be waiting for all spawned processes to complete, including MATLAB's background services, before marking the job as complete. When these processes are started by the runner, they may be considered part of the job's process tree, preventing job completion.