Native Step Runner Integration for Docker Executor

What does this MR do?

This MR adds native step-runner integration to the docker executor. "Native integration" here means the docker executor will run steps via the step-runner gRPC service.

The bulk of this code is the Dialer implementation that tunnels gRPC request through a text-based docker exec call. This code is absolutely heinous, and mostly involves reconciling the type required by grpc.WithContextDialer() and the type returned by ContainerExecAttach(). This code will eventually be moved to a new step-runner-client project, but it lived here for now.

The rest of this code is plumbing; detecting when native steps integration is enables and adjusting the execution path accordingly to use the new execution code.

Enabling this feature is done via a single feature flag, FF_USE_NATIVE_STEPS. This flag enables/disables the feature for all executors. We could also do it on a per-executor basis via executor config. I'm amenable to changing the flag name if other's don't like it.

There's also an integration test to run steps jobs. There are a handful of cases there.

Notes:

• I've deliberately kept the changes to the execution path to a minimum to avoid breaking other executors.
• This MR assumes the step-runner binary is in the execution environment
• This MR is best reviewed commit-at-a-time; each commits has lots of explanations.

Why was this MR needed?

To implement

• Create RunRequest payload from CI job (#37413 - closed)
• Execute CI job via Step Runner service - Docker... (#37414 - closed)

What's the best way to test this MR?

There's an integration test in this MR. ALternatively one could run a pipeline like so (see https://gitlab.com/avonbertoldi/test-project/-/tree/step-runner-docker-integration).

Click to expand

stages:
  - test

variables:
  FF_USE_NATIVE_STEPS: true

script steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: script1
      script: pwd
    - name: script2
      script: env
    - name: script3
      script: ls -Rlah --ignore .git ../

local steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: ls
      step: ./steps/ls
      inputs:
        opts: -lah
    - name: local_echo
      step: ./steps/echo
      inputs:
        message: Oh hello jimmy!?

# This jobs requires `privileged = true` on the executor
# I haven't got this job to pass yet
action step:
  stage: test
  services:
    - docker:24.0.5-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: action
      step: ./steps/action

remote step:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: remote_hello_world
      step: "https://gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@91141a6e"
      inputs:
        echo: hello world

What are the relevant issue numbers?

• Closes Create RunRequest payload from CI job (#37413 - closed)
• Closes Execute CI job via Step Runner service - Docker... (#37414 - closed)

docs/executors/docker.md

@@ -1149,7 +1149,7 @@ enabling [a network for each job](#create-a-network-for-each-job).
 ## Native Steps Integration
 
 As of version `v17.6.0`, the Docker executor supports executing [CI Steps](https://docs.gitlab.com/ee/ci/steps/)
-natively via the `gRPC` API provided by [`step-runner`](https://gitlab.com/gitlab-org/step-runner.git).
+natively via the `gRPC` API provided by [`step-runner`](https://gitlab.com/gitlab-org/step-runner).
 
 To enable this mode of execution, in addition to specifying CI jobs using the `run` keyword instead of the legacy
 `script` keyword, you must enable the feature flag `FF_USE_NATIVE_STEPS`. This can be done at the job or pipeline level.
@@ -1181,4 +1181,4 @@ Note that this is temporary, and in the future we will inject a `step-runner` bi
 - Running a step that runs a docker container is subject to the same configuration and limitations as legacy `scripts`.
 Namely, you must use
 [docker-in-docker](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/docs/executors/docker.md?ref_type=heads#use-docker-in-docker-with-privileged-mode).
-- Running [`Github Actions`](https://gitlab.com/gitlab-org/step-runner.git) currently does not work.
+- Running [`Github Actions`](https://gitlab.com/components/actionrunner) currently does not work.