Native Step Runner Integration for Docker Executor
-
Review changes -
-
Download -
Patches
-
Plain diff
What does this MR do?
This MR adds native step-runner integration to the docker executor. "Native integration" here means the docker executor will run steps via the step-runner gRPC service.
The bulk of this code is the Dialer
implementation that tunnels gRPC request through a text-based docker exec
call. This code is absolutely heinous, and mostly involves reconciling the type required by grpc.WithContextDialer()
and the type returned by ContainerExecAttach()
. This code will eventually be moved to a new step-runner-client
project, but it lived here for now.
The rest of this code is plumbing; detecting when native steps integration is enables and adjusting the execution path accordingly to use the new execution code.
Enabling this feature is done via a single feature flag, FF_USE_NATIVE_STEPS
. This flag enables/disables the feature for all executors. We could also do it on a per-executor basis via executor config. I'm amenable to changing the flag name if other's don't like it.
There's also an integration test to run steps jobs. There are a handful of cases there.
Notes:
- I've deliberately kept the changes to the execution path to a minimum to avoid breaking other executors.
- This MR assumes the step-runner binary is in the execution environment
- This MR is best reviewed commit-at-a-time; each commits has lots of explanations.
Why was this MR needed?
To implement
- Create RunRequest payload from CI job (#37413 - closed)
- Execute CI job via Step Runner service - Docker... (#37414 - closed)
What's the best way to test this MR?
There's an integration test in this MR. ALternatively one could run a pipeline like so (see https://gitlab.com/avonbertoldi/test-project/-/tree/step-runner-docker-integration).
Click to expand
stages:
- test
variables:
FF_USE_NATIVE_STEPS: true
script steps:
stage: test
image:
name: registry.gitlab.com/gitlab-org/step-runner:v0
run:
- name: script1
script: pwd
- name: script2
script: env
- name: script3
script: ls -Rlah --ignore .git ../
local steps:
stage: test
image:
name: registry.gitlab.com/gitlab-org/step-runner:v0
run:
- name: ls
step: ./steps/ls
inputs:
opts: -lah
- name: local_echo
step: ./steps/echo
inputs:
message: Oh hello jimmy!?
# This jobs requires `privileged = true` on the executor
# I haven't got this job to pass yet
action step:
stage: test
services:
- docker:24.0.5-dind
variables:
DOCKER_HOST: tcp://docker:2376
image:
name: registry.gitlab.com/gitlab-org/step-runner:v0
run:
- name: action
step: ./steps/action
remote step:
stage: test
image:
name: registry.gitlab.com/gitlab-org/step-runner:v0
run:
- name: remote_hello_world
step: "https://gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@91141a6e"
inputs:
echo: hello world
What are the relevant issue numbers?
Merge request reports
- version 30b6b248fe
- version 293a9288e6
- version 284a4245bf
- version 2750ad7704
- version 26dbff81c0
- version 25e03b3770
- version 24244d75fb
- version 23d99b3cec
- version 224b69d66e
- version 212ad95b59
- version 20d8cf6d1b
- version 1994b933aa
- version 182fedc316
- version 1712a9d8aa
- version 16fcf24754
- version 15fa3703cf
- version 14b9059c08
- version 137950f951
- version 12131d61f5
- version 11fe300f39
- version 10d0390376
- version 975f7e9f7
- version 8fc25f168
- version 7b07f061c
- version 6e92fec06
- version 508793dae
- version 422dd2f8a
- version 340fc133c
- version 2513f3ef4
- version 1513f3ef4
- main (base)
- latest version4f75f6f318 commits,
- version 30b6b248fe18 commits,
- version 293a9288e618 commits,
- version 284a4245bf18 commits,
- version 2750ad770418 commits,
- version 26dbff81c018 commits,
- version 25e03b377018 commits,
- version 24244d75fb18 commits,
- version 23d99b3cec18 commits,
- version 224b69d66e18 commits,
- version 212ad95b5917 commits,
- version 20d8cf6d1b17 commits,
- version 1994b933aa17 commits,
- version 182fedc31617 commits,
- version 1712a9d8aa17 commits,
- version 16fcf2475422 commits,
- version 15fa3703cf23 commits,
- version 14b9059c0823 commits,
- version 137950f95122 commits,
- version 12131d61f522 commits,
- version 11fe300f3922 commits,
- version 10d039037617 commits,
- version 975f7e9f717 commits,
- version 8fc25f16814 commits,
- version 7b07f061c13 commits,
- version 6e92fec0613 commits,
- version 508793dae13 commits,
- version 422dd2f8a13 commits,
- version 340fc133c13 commits,
- version 2513f3ef413 commits,
- version 1513f3ef413 commits,
- Side-by-side
- Inline
There are no changes yet
No changes between version 26 and version 26