Skip to content

Instance autoscale executor: Shell is not login shell?

Summary

I'm trying to set up autoscaled runners (with AWS) using the instance executor. Jobs submitted to this runner need to be able to perform Docker builds, so this is why I'm using instance instead of docker-autoscaler (giving the job direct access to the host is much easier to deal with than trying to get Docker-in-Docker to work nicely).

The AMI the autoscaled instances use has several utilities pre-installed for jobs to use. One such utility is Poetry. I installed Poetry under /etc/poetry so all users have access to it. As a result, I have to add that directory to PATH. I added

export PATH=$PATH:/etc/poetry/bin

to /home/gitlab-runner/.profile to do this.

When I log in to an instance using this AMI as gitlab-runner, I am able to run poetry. However, when I submit a job that needs to use poetry, I get poetry: command not found. If I echo PATH in the CI/CD script, I can see that /etc/poetry/bin is not on PATH. This makes me think the instance executor is not running the script using a login shell.

I can work around this by manually sourcing .profile by running source ~/.profile as the first command in the CI/CD script. But I don't find this quite ideal...

I'm curious if this is intended behavior, or a bug? Further, is there a way to configure the executor to use a login shell?

I understand the instance executor is experimental, so it could be that this is just how things must be for now. 😞

Steps to reproduce

Run a job on a runner using the instance executor that requires something in ~/.profile to be sourced in order to succeed.

.gitlab-ci.yml 
deploy:
  stage: deploy
  # (commented-out while debugging)
  # rules:
  # - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  needs: []
  tags:
  - docker-build  # tells the job to run using the `instance` runner I have set up
  script:
  - echo $PATH
  - VERSION=$(poetry version -s)
  - 'echo "Plugin version: $VERSION"'
  - docker build -t "my-plugin:$VERSION" .

Actual behavior

The job crashes on the VERSION=$(poetry version -s) command with bash: line 137: poetry: command not found.

Expected behavior

VERSION=$(poetry version -s) should succeed (or at least not crash due to poetry not being found) and the rest of the job should run.

Relevant logs and/or screenshots

job log 
Running with gitlab-runner 16.2.1 (674e0e29)
  on Autoscale (Docker build) x8AJsgEyR, system ID: s_53d66bad37ca
Resolving secrets
Preparing the "instance" executor
Preparing instance...
Dialing instance i-0fa62a46e31b6d14d...
Instance i-0fa62a46e31b6d14d connected
Preparing environment
Running on ip-172-16-1-93 via gitlab-runner-manager...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /home/ubuntu/builds/x8AJsgEyR/0/REDACTED/.git/
Created fresh repository.
Checking out 050e14ac as detached HEAD (ref is test-docker-autoscaling-ci)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
$ VERSION=$(poetry version -s)
bash: line 137: poetry: command not found
Cleaning up project directory and file based variables
ERROR: Job failed: Process exited with status 1

Environment description

  • Our GitLab instance is self-hosted, so the runners are also self-hosted.
  • As mentioned above, I have a runner manager running on an EC2 instance, with the relevant runner configured to use the instance executor.
config.toml contents 
concurrent = 40
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

# ...
# Here, three other autoscale-related [[runners]] sections using the `docker-autoscaler` executor
# I believe these are irrelevant to the issue
# ...

[[runners]]
  name = "Autoscale (Docker build)"
  url = "REDACTED"
  id = 32
  token = "REDACTED"
  token_obtained_at = 2023-08-17T21:38:07Z
  token_expires_at = 0001-01-01T00:00:00Z
  shell = "bash"
  executor = "instance"
  [runners.cache]
    Type = "s3"
    Path = "gitlab-cache"
    Shared = true
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "REDACTED"
      SecretKey = "REDACTED"
      BucketName = "REDACTED"
      BucketLocation = "us-east-1"
  [runners.autoscaler]
    capacity_per_instance = 1
    max_use_count = 1
    max_instances = 10
    plugin = "fleeting-plugin-aws"
    [runners.autoscaler.plugin_config]
      name = "GitLab_runner_fleet_docker-build"
    [runners.autoscaler.connector_config]
      username = "ubuntu"
      password = ""
      key_path = ""
      use_static_credentials = false
      keepalive = "0s"
      timeout = "0s"
      use_external_addr = true

    [[runners.autoscaler.policy]]
      periods = ["* 12-22 * * MON-FRI"]
      idle_count = 1
      idle_time = "20m0s"
      scale_factor = 0.0
      scale_factor_limit = 0

    [[runners.autoscaler.policy]]
      periods = ["* * * * SAT,SUN", "* 0-11,23 * * MON-FRI"]
      idle_count = 0
      idle_time = "20m0s"
      scale_factor = 0.0

Used GitLab Runner version

Version:      16.2.1
Git revision: 674e0e29
Git branch:   16-2-stable
GO version:   go1.20.5
Built:        2023-08-08T00:08:26+0000
OS/Arch:      linux/amd64