Enable GitLab Functions on the shell executor

Arran Walkerrequested to merge
ajwalker/concrete-shell-executor into main

What does this MR do?

Runs native steps (the run: keyword, and the FF_CONCRETE script migration) on the shell executor through an in-process step-runner instead of the abstract shell.

How it works

One step-runner gRPC server runs per runner process, listening on a generated unix socket. It starts lazily on the first Connect, is shared by every shell build (concurrent builds are isolated by step-runner's per-job request id), respawns if it exits unexpectedly, and stops on runner shutdown via the ManagedExecutorProvider hook. Only the concrete builtin is registered: shell dispatches the whole job as a single builtin://concrete step, so the per-stage script_legacy migration is not wired up.

The server standup lives in a new steps/localserver package, which the upcoming custom executor MR reuses for the helper's one-shot steps run command. The steps serve command predates the package and is intentionally not migrated here (it carries extra behavior: ready message, script_legacy, child process management); that is a follow-up.

Dispatch gating

Shell can only run native steps through the concrete whole-job path, so a run: job without FF_CONCRETE cannot work. Rather than hardcoding executor names in common, executors declare a new FeaturesInfo.NativeStepsViaConcreteOnly capability (not serialized to GitLab, since FF_CONCRETE is a per-job flag the server cannot know when routing). Build.executeScript rejects native-steps jobs on such executors before any stage runs, classified as runner_configuration_error rather than a retryable runner_system_failure.

Kubernetes has the same constraint and declares the same capability. Its run: jobs without FF_CONCRETE previously created the pod, cloned sources, and only failed at Connect; they now fail fast before any stage runs, with the same error. The Connect check remains as defense-in-depth.

UseNativeSteps() now excludes Windows only for the hybrid per-stage path, since the concrete path works on Windows.

Behavior changes

  • Shell advertises NativeStepsIntegration, so GitLab now routes run: jobs to shell runners. Without FF_CONCRETE they fail immediately with native step execution (e.g. the 'run:' keyword) on this executor requires the FF_CONCRETE feature flag to be enabled. This matches the posture kubernetes already ships.
  • Kubernetes run: jobs without FF_CONCRETE fail fast (no pod created) instead of failing mid-job.
  • The rejection reports runner_configuration_error where GitLab supports it, falling back to script_failure via the failure reason compatibility map. Previously an equivalent failure reported runner_system_failure, which is retryable.

Why was this MR needed?

The shell executor had no native steps support. Concrete gives it whole-job step-runner execution without porting the abstract shell's per-stage generation, and is the same path the Docker and Kubernetes executors already use with FF_CONCRETE. The capability-based gate replaces what would otherwise be executor-name checks in common and gives kubernetes fail-fast behavior it previously lacked.

What's the best way to test this MR?

Unit tests cover the gate predicate, the localserver lifecycle (including the $TMPDIR fallback), and the shell provider wiring.

End to end with a shell runner against any GitLab (or runner-backbench):

  1. run: job with FF_CONCRETE=true: executes through the in-process step-runner (trace shows a single Executing "concrete" stage).
  2. Plain script job with FF_CONCRETE=true: whole job dispatches through concrete.
  3. run: job without FF_CONCRETE: fails immediately with the FF_CONCRETE error, no stages run.
  4. Plain script job without FF_CONCRETE: unchanged classic path.

For kubernetes, verify run: + FF_CONCRETE=true still works and run: without it fails before pod creation.

What are the relevant issue numbers?

Closes step-runner#403 (closed)

Merge request reports