Add native steps support to the instance executor

Georgi N. Georgiev | GitLabrequested to merge
instance-steps into main

What does this MR do?

Enables GitLab Functions (native steps, the run keyword) on the instance executor, mirroring the docker executor's integration in executors/docker/steps.go.

The instance executor now implements steps.Connector: it starts step-runner on the instance (steps serve) and returns a dialer that connects to it (steps proxy) over the connector. The autoscaler wrapper already delegates Connect to the inner executor, so no autoscaler changes were needed.

Key details

  • Per-job socket (/tmp/step-runner-<jobID>.sock) avoids collisions between concurrent jobs on a shared VM (capacity_per_instance > 1) and stays within the unix sun_path length limit.
  • Lifecycle: step-runner is tied to a stdin keepalive so it is reliably terminated on connection teardown or context cancellation (the connector cannot stop a remote process directly), and a stale socket is cleared before bind.
  • Ready detection: the ready marker is redirected to stdout (2>&1) so it is detected regardless of whether the connector separates stdout/stderr.
  • Binary checks: before serving, the instance's gitlab-runner binary is probed - a clear error is returned when it is missing, native steps on Windows instances are rejected, and a version mismatch with the manager is warned about.

This assumes the gitlab-runner binary is present on the instance image (as documented). Provisioning the binary onto the instance is intentionally out of scope and tracked separately.

Related

Implements step-runner#467.

Testing

  • Unit tests: executors/instance/steps_test.go (connect, missing binary, Windows guard, version mismatch, version parsing).
  • Integration test: TestStepsIntegration (mirrors the docker steps integration test) - passes end-to-end over the SSH stub with run-keyword jobs.
  • Verified locally with glci: go:lint, check test directives, docs:lint markdown all pass.

Docs

Updated docs/executors/instance.md to note that native steps require the GitLab Runner binary on the instance image (Linux only, version-aligned with the manager).

Manual end-to-end test (instance executor, real GCP)

Beyond the unit and integration tests, this MR was verified on a real GCP VM (not the SSH stub used in the integration test). A local runner manager (this branch's gitlab-runner dev build) used the fleeting-plugin-googlecloud plugin to provision a VM, and a run: job executed through the new instance-executor native-steps path. To match the manager and instance step-runner versions exactly, the same dev binary was used on both sides.

This exercises exactly the code in this MR: Connect starting steps serve, the per-job 0700 socket directory, the --version probe (no mismatch warning since manager and instance ran the same build), and steps proxy driving the steps over SSH.

We ran it two ways to confirm the binary-delivery assumption:

Approach A: binary baked into the instance image (recommended, matches the docs)

The instance image already has git and gitlab-runner in PATH, which is the documented expectation for the instance executor. No instance_ready_command is needed because the binary is present the moment the instance is reachable. This is the clean, representative case.

config.toml (no readiness gate):

   [[runners]]
     name = "instance-steps-gcp-test"
     url = "https://gitlab.com"
     token = "glrt-***redacted***"
     executor = "instance"
   
     [runners.autoscaler]
       plugin = "fleeting-plugin-googlecloud"
       capacity_per_instance = 1
       max_use_count = 1
       max_instances = 1
       # Approach B only (see below):
       # instance_ready_command = "timeout 300 bash -c 'until [ -f /run/steps-ready ]; do sleep 2; done'"
   
       [runners.autoscaler.plugin_config]
         name    = "steps-test-mig"
         project = "<gcp-project>"
         zone    = "europe-west4-b"
   
       [runners.autoscaler.connector_config]
         os                = "linux"
         arch              = "amd64"
         protocol          = "ssh"
         username          = "fleeting"
         use_external_addr = true
Job log (Approach A, no instance_ready_command) 
2026-06-09T10:21:22.856010Z 00O Running with gitlab-runner v1.11.1-0.20260602205936-16877607f353+dirty (16877607)
2026-06-09T10:21:22.856041Z 00O   on instance-steps-gcp-test El92-x5Sp, system ID: s_8f980197ad83
2026-06-09T10:21:22.856103Z 00O section_start:1781000482:prepare_executor

2026-06-09T10:21:22.856108Z 00O+Preparing the "instance" executor
2026-06-09T10:21:46.017542Z 00O Preparing instance...
2026-06-09T10:21:46.017890Z 00O Dialing instance https://www.googleapis.com/compute/v1/projects/group-verify-df9383/zones/europe-west4-b/instances/steps-test-mig-c4dbe71eef70e01d...
2026-06-09T10:22:10.107763Z 00O Instance https://www.googleapis.com/compute/v1/projects/group-verify-df9383/zones/europe-west4-b/instances/steps-test-mig-c4dbe71eef70e01d connected
2026-06-09T10:22:10.107805Z 00O section_end:1781000530:prepare_executor

2026-06-09T10:22:10.108642Z 00O+section_start:1781000530:prepare_script

2026-06-09T10:22:10.108653Z 00O+Preparing environment
2026-06-09T10:22:12.804276Z 01O Running on steps-test-mig-c4dbe71eef70e01d via ggeorgiev--20250711-WKRF7...
2026-06-09T10:22:12.812371Z 00O section_end:1781000532:prepare_script

2026-06-09T10:22:12.813462Z 00O+section_start:1781000532:get_sources

2026-06-09T10:22:12.813470Z 00O+Getting source from Git repository
2026-06-09T10:22:14.109565Z 01O Gitaly correlation ID: a08f68b36e54d0f2-SOF
2026-06-09T10:22:14.122009Z 01O Fetching changes with git depth set to 50...
2026-06-09T10:22:14.139823Z 01O Initialized empty Git repository in /home/fleeting/builds/El92-x5Sp/0/ggeorgiev_gitlab/playground/.git/
2026-06-09T10:22:14.145534Z 01O Created fresh repository.
2026-06-09T10:22:21.257005Z 01O Checking out 5a1a401a as detached HEAD (ref is instance-steps-test)...
2026-06-09T10:22:21.797149Z 01O Skipping Git submodules setup
2026-06-09T10:22:21.797306Z 00O section_end:1781000541:get_sources

2026-06-09T10:22:21.798781Z 00O+section_start:1781000541:step_run

2026-06-09T10:22:21.798798Z 00O+Executing "step_run" stage of the job script
2026-06-09T10:22:27.606613Z 01O step-runner is listening on socket /tmp/step-runner-14763539699/step-runner.sock
2026-06-09T10:22:27.990992Z 02E Running step name=hello
2026-06-09T10:22:27.993675Z 01O hello from native steps on the instance executor
2026-06-09T10:22:27.994513Z 02E Running step name=host_info
2026-06-09T10:22:27.996567Z 01O runner=
2026-06-09T10:22:27.998030Z 01O hostname=steps-test-mig-c4dbe71eef70e01d
2026-06-09T10:22:28.023908Z 01O whoami=fleeting
2026-06-09T10:22:28.024932Z 02E Running step name=git_version
2026-06-09T10:22:28.029415Z 01O git version 2.34.1
2026-06-09T10:22:28.030253Z 02E Running step name=runner_version
2026-06-09T10:22:28.137730Z 01O Version:      v1.11.1-0.20260602205936-16877607f353+dirty
2026-06-09T10:22:28.137766Z 01O Git revision: 16877607
2026-06-09T10:22:28.227888Z 00O section_end:1781000548:step_run

2026-06-09T10:22:28.229048Z 00O+section_start:1781000548:cleanup_file_variables

2026-06-09T10:22:28.229060Z 00O+Cleaning up project directory and file based variables
2026-06-09T10:22:28.464566Z 00O section_end:1781000548:cleanup_file_variables

2026-06-09T10:22:28.465344Z 00O Job succeeded

Approach B: install the binary at boot via a startup script

A convenience for testing without baking an image. The startup script installs git and downloads the binary on boot:

   #!/bin/bash
   set -x
   export DEBIAN_FRONTEND=noninteractive
   apt-get update -y && apt-get install -y git curl ca-certificates
   # fetch the version-matched gitlab-runner from GCS using the VM service account
   TOKEN=$(curl -s -H "Metadata-Flavor: Google" \
     "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
     | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
   curl -fsS -H "Authorization: Bearer ${TOKEN}" -o /usr/bin/gitlab-runner \
     "https://storage.googleapis.com/storage/v1/b/<bucket>/o/gitlab-runner?alt=media"
   chmod +x /usr/bin/gitlab-runner
   touch /run/steps-ready

Because that install runs asynchronously and the instance is reported ready as soon as SSH answers, the binary may not exist yet when the job calls steps serve. In that case we add a readiness gate so the job waits for the install to finish:

    instance_ready_command = "timeout 300 bash -c 'until [ -f /run/steps-ready ]; do sleep 2; done'"

This gate is not part of the feature and not something step-runner handles; it is a generic, pre-existing autoscaler option that only compensates for the async boot-install in this approach. With Approach A it is unnecessary.

Job log (Approach B, boot-install + gate) 
2026-06-09T09:56:16.624173Z 00O Running with gitlab-runner v1.11.1-0.20260602205936-16877607f353+dirty (16877607)
2026-06-09T09:56:16.624207Z 00O   on instance-steps-gcp-test El92-x5Sp, system ID: s_8f980197ad83
2026-06-09T09:56:16.624271Z 00O section_start:1780998976:prepare_executor

2026-06-09T09:56:16.624275Z 00O+Preparing the "instance" executor
2026-06-09T09:57:31.975271Z 00O Preparing instance...
2026-06-09T09:57:31.975495Z 00O Dialing instance https://www.googleapis.com/compute/v1/projects/group-verify-df9383/zones/europe-west4-b/instances/steps-test-mig-fc84966cebae6437...
2026-06-09T09:57:32.408946Z 00O Instance https://www.googleapis.com/compute/v1/projects/group-verify-df9383/zones/europe-west4-b/instances/steps-test-mig-fc84966cebae6437 connected
2026-06-09T09:57:32.409010Z 00O section_end:1780999052:prepare_executor

2026-06-09T09:57:32.409889Z 00O+section_start:1780999052:prepare_script

2026-06-09T09:57:32.409892Z 00O+Preparing environment
2026-06-09T09:57:32.664784Z 01O Running on steps-test-mig-fc84966cebae6437 via ggeorgiev--20250711-WKRF7...
2026-06-09T09:57:32.667432Z 00O section_end:1780999052:prepare_script

2026-06-09T09:57:32.668482Z 00O+section_start:1780999052:get_sources

2026-06-09T09:57:32.668490Z 00O+Getting source from Git repository
2026-06-09T09:57:32.911075Z 01O Gitaly correlation ID: a08f43ee8fa4d0db-SOF
2026-06-09T09:57:32.922157Z 01O Fetching changes with git depth set to 50...
2026-06-09T09:57:32.931108Z 01O Initialized empty Git repository in /home/fleeting/builds/El92-x5Sp/0/ggeorgiev_gitlab/playground/.git/
2026-06-09T09:57:32.936950Z 01O Created fresh repository.
2026-06-09T09:57:39.396119Z 01O Checking out 5a1a401a as detached HEAD (ref is instance-steps-test)...
2026-06-09T09:57:40.009876Z 01O Skipping Git submodules setup
2026-06-09T09:57:40.010078Z 00O section_end:1780999060:get_sources

2026-06-09T09:57:40.013219Z 00O+section_start:1780999060:step_run

2026-06-09T09:57:40.013231Z 00O+Executing "step_run" stage of the job script
2026-06-09T09:57:40.442587Z 01O step-runner is listening on socket /tmp/step-runner-14762993586/step-runner.sock
2026-06-09T09:57:40.681991Z 02E Running step name=hello
2026-06-09T09:57:40.684756Z 01O hello from native steps on the instance executor
2026-06-09T09:57:40.685578Z 02E Running step name=host_info
2026-06-09T09:57:40.687800Z 01O runner=
2026-06-09T09:57:40.689248Z 01O hostname=steps-test-mig-fc84966cebae6437
2026-06-09T09:57:40.693728Z 01O whoami=fleeting
2026-06-09T09:57:40.694620Z 02E Running step name=git_version
2026-06-09T09:57:40.698201Z 01O git version 2.34.1
2026-06-09T09:57:40.699005Z 02E Running step name=runner_version
2026-06-09T09:57:40.807652Z 01O Version:      v1.11.1-0.20260602205936-16877607f353+dirty
2026-06-09T09:57:40.807688Z 01O Git revision: 16877607
2026-06-09T09:57:40.969130Z 00O section_end:1780999060:step_run

2026-06-09T09:57:40.969647Z 00O+section_start:1780999060:cleanup_file_variables

2026-06-09T09:57:40.969650Z 00O+Cleaning up project directory and file based variables
2026-06-09T09:57:41.120315Z 00O section_end:1780999061:cleanup_file_variables

2026-06-09T09:57:41.120490Z 00O Job succeeded

Common setup (both approaches)

  1. Cross-compile the branch binary for the VM: GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o gitlab-runner .
  2. Stage it where the VM can read it (GCS bucket in the test project): gcloud storage cp gitlab-runner gs://<bucket>/gitlab-runner. Approach A bakes it into a custom image; Approach B downloads it at boot.
  3. Build the GCP fleeting plugin into PATH (fleeting-plugin-googlecloud).
  4. Create a zonal managed instance group (size 0, autoscaling off) referenced by the config.
  5. Register a project runner (instance executor) and run the manager: gitlab-runner run -c config.toml. It receives the job, scales the MIG to 1, connects over SSH, runs the steps, and tears the instance down afterwards (max_use_count = 1).

Note: provisioning the binary onto the instance is intentionally out of scope for this MR and tracked separately. Both approaches above are just ways to satisfy the documented "binary present on the instance" expectation for the test.

The job logs show the per-job 0700 socket directory keyed on the real job ID, the matched runner version (no mismatch warning), and all steps running to Job succeeded.

Closes step-runner#408 (closed)

Edited by Arran Walker

Merge request reports