Native Step Runner Integration for Docker Executor

What does this MR do?

This MR adds native step-runner integration to the docker executor. "Native integration" here means the docker executor will run steps via the step-runner gRPC service.

The bulk of this code is the Dialer implementation that tunnels gRPC request through a text-based docker exec call. This code is absolutely heinous, and mostly involves reconciling the type required by grpc.WithContextDialer() and the type returned by ContainerExecAttach(). This code will eventually be moved to a new step-runner-client project, but it lived here for now.

The rest of this code is plumbing; detecting when native steps integration is enables and adjusting the execution path accordingly to use the new execution code.

Enabling this feature is done via a single feature flag, FF_USE_NATIVE_STEPS. This flag enables/disables the feature for all executors. We could also do it on a per-executor basis via executor config. I'm amenable to changing the flag name if other's don't like it.

There's also an integration test to run steps jobs. There are a handful of cases there.

Notes:

• I've deliberately kept the changes to the execution path to a minimum to avoid breaking other executors.
• This MR assumes the step-runner binary is in the execution environment
• This MR is best reviewed commit-at-a-time; each commits has lots of explanations.

Why was this MR needed?

To implement

• Create RunRequest payload from CI job (#37413 - closed)
• Execute CI job via Step Runner service - Docker... (#37414 - closed)

What's the best way to test this MR?

There's an integration test in this MR. ALternatively one could run a pipeline like so (see https://gitlab.com/avonbertoldi/test-project/-/tree/step-runner-docker-integration).

Click to expand

stages:
  - test

variables:
  FF_USE_NATIVE_STEPS: true

script steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: script1
      script: pwd
    - name: script2
      script: env
    - name: script3
      script: ls -Rlah --ignore .git ../

local steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: ls
      step: ./steps/ls
      inputs:
        opts: -lah
    - name: local_echo
      step: ./steps/echo
      inputs:
        message: Oh hello jimmy!?

# This jobs requires `privileged = true` on the executor
# I haven't got this job to pass yet
action step:
  stage: test
  services:
    - docker:24.0.5-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: action
      step: ./steps/action

remote step:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: remote_hello_world
      step: "https://gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@91141a6e"
      inputs:
        echo: hello world

What are the relevant issue numbers?

• Closes Create RunRequest payload from CI job (#37413 - closed)
• Closes Execute CI job via Step Runner service - Docker... (#37414 - closed)

.gitlab/ci/postrelease.gitlab-ci.yml

@@ -16,9 +16,12 @@ trigger runner-incept tests:
     - "binaries: [linux/amd64 linux/arm64]"
     - "binaries: [linux/s390x linux/ppc64le]"
     - "helper images: [alpine]"
+    - "development docker images"
+    - "development helper docker images"
   stage: postrelease
   variables:
     UPSTREAM_CI_COMMIT_REF: $UPSTREAM_CI_COMMIT_REF
+    UPSTREAM_CI_COMMIT_SHORT_SHA: $CI_COMMIT_SHORT_SHA
   trigger:
     project: gitlab-org/ci-cd/tests/runner-incept
     # strategy: depend