Native Step Runner Integration for Docker Executor

What does this MR do?

This MR adds native step-runner integration to the docker executor. "Native integration" here means the docker executor will run steps via the step-runner gRPC service.

The bulk of this code is the Dialer implementation that tunnels gRPC request through a text-based docker exec call. This code is absolutely heinous, and mostly involves reconciling the type required by grpc.WithContextDialer() and the type returned by ContainerExecAttach(). This code will eventually be moved to a new step-runner-client project, but it lived here for now.

The rest of this code is plumbing; detecting when native steps integration is enables and adjusting the execution path accordingly to use the new execution code.

Enabling this feature is done via a single feature flag, FF_USE_NATIVE_STEPS. This flag enables/disables the feature for all executors. We could also do it on a per-executor basis via executor config. I'm amenable to changing the flag name if other's don't like it.

There's also an integration test to run steps jobs. There are a handful of cases there.

Notes:

• I've deliberately kept the changes to the execution path to a minimum to avoid breaking other executors.
• This MR assumes the step-runner binary is in the execution environment
• This MR is best reviewed commit-at-a-time; each commits has lots of explanations.

Why was this MR needed?

To implement

• Create RunRequest payload from CI job (#37413 - closed)
• Execute CI job via Step Runner service - Docker... (#37414 - closed)

What's the best way to test this MR?

There's an integration test in this MR. ALternatively one could run a pipeline like so (see https://gitlab.com/avonbertoldi/test-project/-/tree/step-runner-docker-integration).

Click to expand

stages:
  - test

variables:
  FF_USE_NATIVE_STEPS: true

script steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: script1
      script: pwd
    - name: script2
      script: env
    - name: script3
      script: ls -Rlah --ignore .git ../

local steps:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: ls
      step: ./steps/ls
      inputs:
        opts: -lah
    - name: local_echo
      step: ./steps/echo
      inputs:
        message: Oh hello jimmy!?

# This jobs requires `privileged = true` on the executor
# I haven't got this job to pass yet
action step:
  stage: test
  services:
    - docker:24.0.5-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: action
      step: ./steps/action

remote step:
  stage: test
  image:
    name: registry.gitlab.com/gitlab-org/step-runner:v0
  run:
    - name: remote_hello_world
      step: "https://gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@91141a6e"
      inputs:
        echo: hello world

What are the relevant issue numbers?

• Closes Create RunRequest payload from CI job (#37413 - closed)
• Closes Execute CI job via Step Runner service - Docker... (#37414 - closed)

dockerfiles/ci/ubi.fips.base.Dockerfile

@@ -3,22 +3,36 @@ ARG UBI_MINIMAL_IMAGE=redhat/ubi9-minimal
 
 FROM ${UBI_MICRO_IMAGE} AS initial
 
+# Download the latest git-lfs for this distro from upstream's rpm repo since ubi9's version is always several releases
+# behind. We'll install it in the next layer.
+FROM ${UBI_MINIMAL_IMAGE} AS git-lfs
+RUN microdnf install --best --refresh --assumeyes --nodocs --setopt=install_weak_deps=0 --setopt=tsflags=nodocs yum
+RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh | bash
+RUN yum download git-lfs
+RUN mv git-lfs*.rpm git-lfs.rpm
+
 FROM ${UBI_MINIMAL_IMAGE} AS build
-ARG DNF_OPTS_ROOT="--installroot=/install-root/ --noplugins  --setopt=reposdir=/install-root/etc/yum.repos.d/ \
+ARG DNF_OPTS_ROOT="--installroot=/install-root/ --noplugins --setopt=reposdir=/install-root/etc/yum.repos.d/ \
     --setopt=varsdir=/install-var/ --config= --setopt=cachedir=/install-cache/"
 
 RUN mkdir -p /install-root/ /install-var
 COPY --from=initial / /install-root/
 
-# these packages are required by downstream images, but we don't know anymore specifically which images require which
+# These packages are required by downstream images, but we don't know anymore specifically which images require which
+# packages.
 RUN microdnf update ${DNF_OPTS_ROOT} --best --refresh --assumeyes --nodocs --setopt=install_weak_deps=0 --setopt=tsflags=nodocs
-RUN microdnf install ${DNF_OPTS_ROOT} --best --refresh  --assumeyes --nodocs  --setopt=install_weak_deps=0 --setopt=tsflags=nodocs \
-        hostname procps tar gzip ca-certificates tzdata openssl git git-lfs findutils
-RUN microdnf clean  ${DNF_OPTS_ROOT}  all \
+RUN microdnf install ${DNF_OPTS_ROOT} --best --refresh --assumeyes --nodocs --setopt=install_weak_deps=0 --setopt=tsflags=nodocs \
+        hostname procps tar gzip ca-certificates tzdata openssl git findutils
+
+# Install git-lfs from upstream's rpm package
+COPY --from=git-lfs /git-lfs.rpm .
+RUN rpm -i --root=/install-root/ ./git-lfs.rpm
+
+RUN microdnf clean ${DNF_OPTS_ROOT} all \
     && rm -f /install-root/var/lib/dnf/history*
 
 FROM ${UBI_MICRO_IMAGE} AS final
 
-COPY --from=build  /install-root/ /
+COPY --from=build /install-root/ /
 RUN git-lfs install --skip-repo \
     && rm -f /var/lib/dnf/history*