# Enable GitLab Functions on shell executor ## Summary Add native steps (GitLab Functions) support to the shell executor, mirroring the docker executor's integration in `executors/docker/steps.go` (ready marker, serve, proxy) — but adapted for local-host execution. Because the shell executor runs on the same machine as `gitlab-runner` itself, the connector can spawn `step-runner serve` as a local subprocess and dial its unix socket directly. No transport shim, no proxy subprocess, no remote bootstrap. ## Lifecycle decision: one shared, runner-managed subprocess Rather than spawning one `step-runner serve` per job (the model used for docker/instance executors today), the shell executor will run **one shared `step-runner serve` subprocess per gitlab-runner runtime**, owned and torn down by gitlab-runner itself. To be clear up front: **this is not daemon mode**. A real daemon would be started by the user (systemd unit or hand-rolled), live independently of any gitlab-runner process, listen on a published well-known socket path, and enable resumability across runner restarts. None of that is delivered here. What this iteration does deliver is a long-lived shared subprocess under runner control, which incidentally exercises some of the same characteristics a real daemon will need to get right (multi-tenant job isolation, gRPC-stream-close as the job cancel signal, crash blast-radius across in-flight jobs). Treat the wins on those fronts as a side benefit, not the goal. Why share the subprocess at all on the shell executor (when docker/instance spawn per-job)? - The shell executor is the cheapest, lowest-blast-radius place to validate `step-runner`'s multi-tenant story. - Local-only means we can manage the subprocess with stock Go process primitives — no SSH/WinRM hangup edge cases to fight. - Wiring it through `ManagedExecutorProvider.Init` / `Shutdown` (`common/executor.go:94-119`) gives us a clean lifecycle hook that already exists. Behaviour: - One `<runnerCommandPath> steps serve` subprocess, lazily spawned on first `Connect()` and re-used by every subsequent build on this runner. - Per-job cancellation is **not** a process kill. When a build's context is cancelled, the gRPC `RunAndFollow` call returns and that build's gRPC stream closes; `step-runner` server-side cleanup of the job's processes/state is what then takes effect. (Whether that is currently watertight is one of the things this implementation will exercise.) - Crash recovery: if the shared subprocess exits unexpectedly, the next `Connect()` call respawns it. In-flight jobs that were using the dead process fail; new jobs proceed against the new process. - **No job resumability** across runner restarts. When `gitlab-runner` exits, the shared `step-runner serve` is terminated via `Shutdown` (graceful → force, same `helpers/process.NewOSKillWait` machinery `executor.Run` already uses) and any in-flight jobs fail. Resumability is a real-daemon feature, not on offer here. ### Coexistence with a future user-managed daemon When the real daemon ships, the connector will prefer the user-managed daemon over our runner-managed subprocess: - The user-managed daemon listens on a **published well-known path** (e.g. `$XDG_RUNTIME_DIR/step-runner.sock` for user-level, or `/run/step-runner.sock` for system). This path is **distinct** from the auto-created path we use here (see "Socket path" below) so the two can coexist on disk without clashing. - On `Connect()`, the connector tries the well-known path first; only if no daemon is listening does it fall back to spawning/using the runner-managed shared subprocess. - **Ordering caveat to flag in the future docs:** if the user-managed daemon's service starts *after* gitlab-runner, the runner will already have spawned its own subprocess by the time the daemon comes up. The runner will keep using its own for the rest of its lifetime — meaning the user gets the daemon's process running idle alongside ours, and **does not** get resumability. The fix is for users to order their systemd unit before `gitlab-runner.service` (`Before=gitlab-runner.service`, or equivalent). We will not try to live-switch from runner-managed to daemon-managed mid-runtime; the complexity isn't worth it. This iteration only needs to make sure the auto-created path is **distinct** from any published well-known path so future-us has room to add the prefer-daemon logic without disk-level conflicts. ## Feature Flag This feature will be gated behind a new feature flag: **`FF_FUNCTION_MIGRATIONS_ON_SHELL_EXECUTOR`** (added to `helpers/featureflags/flags.go` alongside `UseScriptToStepMigration` and `UseConcrete`). Note: `FF_SCRIPT_TO_STEP_MIGRATION` can be enabled independently of `FF_FUNCTION_MIGRATIONS_ON_SHELL_EXECUTOR`. This allows users to migrate scripts to steps without enabling shell executor functions support, which may be useful for testing or gradual rollout scenarios. **Important:** The shell executor must NOT check this feature flag when a job uses the `run:` syntax. Since GitLab Functions is experimental, gating `run:` usage behind a feature flag is not required. The feature flag only gates the migration path from `script:` to steps. The check in `common/steps.go:UseNativeSteps` already keys off `Job.Run` length OR the migration flags, so the gating belongs at the executor-feature level (see "Enabling" below), not in `UseNativeSteps`. ## Scope / Tasks ### 1. Enabling In `executors/shell/shell.go`, the `featuresUpdater` (lines 179-187) already gates Windows-specific features behind a `runtime.GOOS != "windows"` check. Add the steps integration to the same block: ```go if runtime.GOOS != "windows" { features.NativeStepsIntegration = true // new features.Session = true features.Terminal = true } ``` The Windows guard matches what `common/steps.go:UseNativeSteps` already enforces, and matches the docker windows provider override at `executors/docker/docker_command.go:416`. Add a compile-time interface assertion next to the executor type: ```go var _ steps.Connector = (*executor)(nil) ``` (Same pattern as `executors/internal/autoscaler/executor.go:17` and `executors/docker/machine/machine.go:19`.) The feature-flag gate (`FF_FUNCTION_MIGRATIONS_ON_SHELL_EXECUTOR`) applies only to the script→steps migration path — when the job uses `run:`, no flag check is needed. Gating happens by inspecting `Build.IsFeatureFlagOn(...)` OR `len(Build.Job.Run) > 0` at the point where we decide to advertise/use `NativeStepsIntegration` for migration. ### 2. Provider-owned shared `step-runner serve` The current `NewProvider` returns a `DefaultExecutorProvider`. Wrap it in a new type that owns the shared subprocess and implements `ManagedExecutorProvider`: ```go type stepsProvider struct { executors.DefaultExecutorProvider runnerCommand string mu sync.Mutex server *stepsServer // nil until first Connect; replaced on respawn } func (sp *stepsProvider) Init() { /* lazy: do nothing */ } func (sp *stepsProvider) Shutdown(ctx context.Context, _ *common.Config) { // KillAndWait the shared subprocess if running; remove socket dir. // Honour ctx so we don't block the runner's overall shutdown timeout. } func (sp *stepsProvider) ensureServer() (*stepsServer, error) { sp.mu.Lock() defer sp.mu.Unlock() if sp.server != nil && sp.server.alive() { return sp.server, nil } // (re)spawn: pick socket path, exec runnerCommand, wait for ready marker. s, err := startStepsServer(sp.runnerCommand) if err != nil { return nil, err } sp.server = s return s, nil } ``` `stepsServer` holds the running `process.Commander`, the socket path, a `waitCh` from the `Wait()` goroutine, and an `alive()` helper that returns `false` once the wait channel has fired. Wire-up in `NewProvider`: - Construct the `stepsProvider` with the existing `DefaultExecutorProvider` embedded. - Pass `runnerCommandPath` through. - The executor `creator` closes over `stepsProvider` so `Connect()` can call `sp.ensureServer()`. `ManagedExecutorProvider` is already understood by the runner's lifecycle (see `common/executor.go:94-119`); no wiring changes are needed beyond returning a type that implements it. ### 3. Implement `steps.Connector` (`Connect(ctx) (func() (io.ReadWriteCloser, error), error)`) A new file `executors/shell/steps.go` holds the connector. The flow is short because the heavy lifting lives on the provider: - Call `sp.ensureServer()`. On error, return it (becomes a `BuildError` upstream). - Tee/route any startup stderr produced during a (re)spawn into the **first build that triggered the (re)spawn**'s log. Subsequent builds reuse the already-up server and don't see startup output. Use the existing `executors/internal/readywriter` logic from `executors/docker/steps.go:82` to detect the `"step-runner is listening on socket <path>"` marker emitted by `commands/steps/steps.go:readyMessage`. - Return a closure that, on each invocation, does `net.Dial("unix", server.sockPath)` and returns the resulting `*net.UnixConn` directly. **No proxy subprocess is needed** — the docker/instance executors only need a proxy because they're crossing a process or machine boundary; the shell executor is on the same host and can speak gRPC over the listener directly. `steps.Execute` (`steps/execute.go:53-101`) only needs an `io.ReadWriteCloser` per dial, and `*net.UnixConn` satisfies it. - The dialer closure is invoked once per gRPC connection by the `extended.New` client; `Close()` on the returned conn must close the dialed socket only — it must **not** terminate the shared subprocess. ### 4. Per-job cancellation via gRPC, not process kill This is the key behaviour change vs. the per-job model used by docker/instance: - The build's context flows into `steps.Execute(ctx, …)` (`common/build.go:478-518`) and from there into the gRPC client. When the build is cancelled (graceful) or aborted (hard), the `RunAndFollow` call returns and gRPC closes the stream. - `step-runner` server-side is responsible for tearing down that job's child processes when its stream closes. We rely on this; if it turns out to be flaky, the fix is in `step-runner`, not here. - The shared subprocess keeps running and continues to serve other in-flight builds. ### 5. Socket path Default: `<os.TempDir()>/gitlab-runner-steps-<runnerToken-or-pid>/step-runner.sock`. - A single shared socket per runner process is what we want — every build on this runner dials the same socket. - Disambiguate between coexisting `gitlab-runner` instances on the same host by including a per-runtime suffix (the runner's token short-hash if available, otherwise PID). This avoids collisions when an admin runs multiple runner installs side-by-side. - **Do not** use `api.DefaultSocketPath()` or any other path that a future user-managed daemon might publish. Reserve those well-known paths for the daemon. Our auto-created path must be distinct so the two can coexist on disk when the daemon ships. - Create the parent directory with mode `0o700`. Remove on `Shutdown`. ### 6. Environment isolation This is essentially **free** on the shell executor and worth saying explicitly: - The runner's own `os.Environ()` does **not** contain job variables. Job vars are injected via `export` lines in the generated shell script that's piped into the build shell (see `shells/abstract.go`); they live only inside that shell subprocess. - Therefore, spawning `gitlab-runner steps serve` from the runner process via `process.NewOSCmd` with `Env: os.Environ()` already excludes all `CI_*` / job-level vars by construction. **And** because the subprocess is shared across builds, there is no per-build env to leak in the first place — each build's variables are sent over gRPC inside the `JobInfo` (`steps/execute.go:24-28` and `common/build.go:507-512`). - Action: do not pass `s.Build.GetAllVariables().StringList()` (or anything similar) into the serve command's `Env`. Tests should assert this at the `process.NewOSCmd` boundary. ### 7. Termination plugged into the provider lifecycle - `stepsProvider.Shutdown(ctx, config)` is called by the runner's lifecycle on stop. - It runs `process.NewOSKillWait(logger, graceful, force).KillAndWait(server.cmd, server.waitCh)` against the shared subprocess, then removes the socket dir. - If `ctx` is cancelled before the graceful timeout elapses, `KillAndWait` is bounded by its internal force-kill timeout — we should not block the overall runner shutdown. - The per-build `executor.Cleanup()` does **not** touch the shared subprocess. It only releases the executor's own resources (same as today). ### 8. Logging - Stderr/stdout from the shared subprocess: route to a runner-level log channel by default. During the (re)spawn handshake only, also tee through `readywriter` into the build that triggered the spawn — for that brief window, that build is the closest thing to an "owner" of the new process. - The dialed unix-socket conn carries gRPC frames; do **not** inject it into the build log. Unlike the docker executor, no `omitwriter` is needed here — gRPC owns the conn end-to-end via `steps.Execute`. ### 9. Tests - [ ] Unit tests for `ensureServer`: - first call spawns and returns ready server; - subsequent calls return the same server without respawning; - if `alive()` returns false, next call respawns; - serve exits non-zero before ready → returns `BuildError` with normalized exit code; - context cancel before ready → kill is invoked and `ctx.Err()` is returned. - [ ] Concurrency test: many parallel `Connect()` calls produce one shared server (assert spawn count = 1). - [ ] Per-job cancellation: build A's context cancel does not close build B's gRPC dial against the same shared socket. - [ ] Crash recovery: kill the shared subprocess underneath an idle provider; next `Connect()` respawns; in-flight builds at the moment of crash fail with a meaningful error. - [ ] Environment isolation: assert the `Env` slice passed to `newCommander` is exactly `os.Environ()` (or, more strictly, contains no `CI_*` / `GITLAB_*` keys at the boundary). - [ ] Shutdown: `Shutdown(ctx, cfg)` terminates the subprocess and removes the socket dir; respects `ctx` deadline. - [ ] Optional integration test in `executors/shell/shell_integration_test.go` for two `run:` jobs in sequence sharing one underlying `step-runner serve`. ## Implementation Notes - File layout: - `executors/shell/steps.go` (new): connector + `stepsServer` + `stepsProvider` wrapper. - Minimal edits to `executors/shell/shell.go`: add the `steps.Connector` interface assertion on the executor, update `featuresUpdater`, and have `NewProvider` return the wrapping `stepsProvider`. - Reuse `executors/internal/readywriter` and `helpers/process` rather than reinventing. - The connector wires into the rest of the runner via `common/build.go:573` (`executor.(steps.Connector)`); no changes needed in `common/`. - `gitlab-runner-helper` is **not** required — this executor invokes the same `gitlab-runner` binary that's running, via the `runnerCommandPath` passed to `NewProvider`. - Override-points for tests already exist: `executors/shell/shell.go:20-21` exposes `newProcessKillWaiter` and `newCommander` as package-level vars. Reuse the same indirections in the connector so tests can swap in a fake commander/socket. ## Acceptance Criteria - [ ] Shell executor advertises `NativeStepsIntegration` on non-Windows and runs steps for jobs using the `run:` keyword without any feature flag. - [ ] `FF_FUNCTION_MIGRATIONS_ON_SHELL_EXECUTOR` enables the `script:` → steps migration path on the shell executor. - [ ] Steps connect by dialing a single shared per-runner-process unix socket — no proxy subprocess, no per-job spawn. - [ ] No job variables are present in the environment of the spawned `step-runner serve` process. - [ ] Concurrent builds on the same runner share one `step-runner serve` subprocess. - [ ] Cancelling one build does not affect concurrently-running builds on the same shared subprocess. - [ ] If the shared subprocess crashes, the next build's `Connect()` respawns it. In-flight builds at the moment of crash fail with a meaningful error. - [ ] On runner shutdown, the shared subprocess is terminated and the socket dir is removed. - [ ] Auto-created socket path is distinct from any well-known path a future user-managed daemon would publish, so the two can coexist on disk. - [ ] No job resumability across runner restarts is required (deferred — same stance as the instance executor). - [ ] User-managed daemon (with resumability, well-known socket, and prefer-daemon-over-runner-managed selection) is **not** delivered in this iteration; tracked as a follow-up. - [ ] Tests pass (unit + optional integration).