Commit 76ea814d authored by Tomasz Maczukin's avatar Tomasz Maczukin Committed by Kamil Trzciński
Browse files

Make configuration of helper image more dynamic

parent 0177466a
......@@ -209,7 +209,7 @@ This defines the Docker Container parameters.
| `allowed_services` | Specify wildcard list of services that can be specified in .gitlab-ci.yml. If not present all images are allowed (equivalent to `["*/*:*"]`) |
| `pull_policy` | Specify the image pull policy: `never`, `if-not-present` or `always` (default); read more in the [pull policies documentation](../executors/docker.md#how-pull-policies-work) |
| `sysctls` | specify the sysctl options |
| `helper_image` | [ADVANCED] Override the default helper image used to clone repos and upload artifacts |
| `helper_image` | [ADVANCED] Override the default helper image used to clone repos and upload artifacts. Read the [helper image](#helper-image) section for more details |
Example:
......@@ -631,6 +631,80 @@ Example:
gitlab = "true"
```
## Helper image
When one of `docker`, `docker+machine` or `kubernetes` executors is used, GitLab Runner uses a specific container
to handle Git, artifacts and cache operations. This container is created from a special image, named `helper image`.
The helper image is based on Alpine Linux and it's provided for amd64 and arm architectures. It contains
a `gitlab-runner-helper` binary which is a special compilation of GitLab Runner binary, that contains only a subset
of available commands, as well as git, git-lfs, SSL certificates store and basic configuration of Alpine.
When GitLab Runner is installed from the DEB/RPM packages, both images (`arm64` and `arm` based) are installed on the host.
When the Runner prepares the environment for the job execution, if the image in specified version (based on Runner's git
revision) is not found on Docker Engine, it is automatically loaded. It works like that for both
`docker` and `docker+machine` executors.
Things work a little different for the `kubernetes` executor or when GitLab Runner is installed manually. For manual
installations, the `gitlab-runner-helper` binary is not included and for the `kubernetes` executor,the API of Kubernetes
doesn't allow to load the `gitlab-runner-helper` image from a local archive. In both cases, GitLab Runner will download
the helper image from Docker Hub, from GitLab's official repository `gitlab/gitlab-runner-helper` by using the Runner's
revision and architecture for defining which tag should be downloaded.
### Overriding the helper image
In some cases, you may need to override the helper image. There are many reasons for doing this:
1. **To speed up jobs execution**: In environments with slower internet connection, downloading over and over again the
same image from Docker Hub may generate a significant increase of a job's timings. Downloading the helper image from
a local registry (where the exact copy of `gitlab/gitlab-runner-helper:XYZ` is stored) may speed things up.
1. **Security concerns**: Many people don't like to download external dependencies that were not checked before. There
might be a business rule to use only dependencies that were reviewed and stored in local repositories.
1. **Build environments without internet access**: In some cases, jobs are being executed in an environment which has
a dedicated, closed network (this doesn't apply to the `kubernetes` executor where the image still needs to be downloaded
from an external registry that is available at least to the Kubernetes cluster).
1. **Additional software**: Some users may want to install some additional software to the helper image, like
`openssh` to support submodules accessible via `git+ssh` instead of `git+http`.
In any of the cases described above, it's possible to configure a custom image using the `helper_image` configuration field,
that is available for the `docker`, `docker+machine` and `kubernetes` executors:
```toml
[[runners]]
(...)
executor = "docker"
[runners.docker]
(...)
helper_image = "my.registry.local/gitlab/gitlab-runner-helper:tag"
```
Note that the version of the helper image should be considered as strictly coupled with the version of GitLab Runner.
As it was described above, one of the main reasons of providing such images is that Runner is using the
`gitlab-runner-helper` binary, and this binary is compiled from part of GitLab Runner sources which is using an internal
API that is expected to be the same in both binaries.
The Runner by default references to a `gitlab/gitlab-runner-helper:XYZ` image, where `XYZ` is based
on the Runner's architecture and git revision. Starting with **GitLab Runner 11.3** it's possible to define the version
of used image automatically, by using one of the
[version variables](https://gitlab.com/gitlab-org/gitlab-runner/blob/11-3-stable/common/version.go#L48-50):
```toml
[[runners]]
(...)
executor = "docker"
[runners.docker]
(...)
helper_image = "my.registry.local/gitlab/gitlab-runner-helper:${CI_RUNNER_REVISION}"
```
With that configuration, GitLab Runner will instruct the executor to use the image in version `${CI_RUNNER_REVISION}`,
which is based on its compilation data. After updating the Runner to a new version, this will ensure that the
Runner will try to download the proper image. This of course means that the image should be uploaded to the registry
before upgrading the Runner, otherwise the jobs will start failing with a "No such image" error.
## Note
If you'd like to deploy to multiple servers using GitLab CI, you can create a
......
......@@ -77,7 +77,7 @@ The following keywords help to define the behaviour of the Runner within Kuberne
- `pull_policy`: specify the image pull policy: `never`, `if-not-present`, `always`. The cluster default will be used if not set.
- `node_selector`: A `table` of `key=value` pairs of `string=string`. Setting this limits the creation of pods to kubernetes nodes matching all the `key=value` pairs
- `image_pull_secrets`: A array of secrets that are used to authenticate docker image pulling
- `helper_image`: [ADVANCED] Override the default helper image used to clone repos and upload artifacts
- `helper_image`: [ADVANCED] Override the default helper image used to clone repos and upload artifacts. Read the [helper image][advanced-configuration-helper-image] section of _advanced configuration_ page for more details
- `terminationGracePeriodSeconds`: Duration after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal
- `poll_interval`: How frequently, in seconds, the runner will poll the Kubernetes pod it has just created to check its status. [Default: 3]
- `poll_timeout`: The amount of time, in seconds, that needs to pass before the runner will timeout attempting to connect to the container it has just created (useful for queueing more builds that the cluster can handle at a time) [Default: 180]
......@@ -370,3 +370,4 @@ on other nodes.
[k8s-secret-volume-docs]: https://kubernetes.io/docs/concepts/storage/volumes/#secret
[k8s-config-map-docs]: https://kubernetes.io/docs/tasks/configure-pod-container/configmap/
[k8s-empty-dir-volume-docs]:https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
[advanced-configuration-helper-image]: ../configuration/advanced-configuration.md#helper-image
......@@ -288,6 +288,8 @@ func (e *executor) loadPrebuiltImage(path, ref, tag string) (*types.ImageInspect
func (e *executor) getPrebuiltImage() (*types.ImageInspect, error) {
if imageNameFromConfig := e.Config.Docker.HelperImage; imageNameFromConfig != "" {
imageNameFromConfig = common.AppVersion.Variables().ExpandValue(imageNameFromConfig)
e.Debugln("Pull configured helper_image for predefined container instead of import bundled image", imageNameFromConfig, "...")
return e.getDockerImage(imageNameFromConfig)
}
......
......@@ -2,6 +2,7 @@ package docker
import (
"bytes"
"errors"
"flag"
"fmt"
"io/ioutil"
......@@ -306,6 +307,35 @@ func TestDockerForExistingImage(t *testing.T) {
assert.NotNil(t, image)
}
func TestHelperImageWithVariable(t *testing.T) {
c := new(docker_helpers.MockClient)
defer c.AssertExpectations(t)
c.On("ImageInspectWithRaw", mock.Anything, "gitlab/gitlab-runner:HEAD").
Return(types.ImageInspect{}, nil, errors.New("not found")).
Once()
c.On("ImagePullBlocking", mock.Anything, "gitlab/gitlab-runner:HEAD", mock.Anything).
Return(nil).
Once()
c.On("ImageInspectWithRaw", mock.Anything, "gitlab/gitlab-runner:HEAD").
Return(types.ImageInspect{ID: "helper-image"}, nil, nil).
Once()
e := executor{
client: c,
}
e.Config = common.RunnerConfig{}
e.Config.Docker = &common.DockerConfig{
HelperImage: "gitlab/gitlab-runner:${CI_RUNNER_REVISION}",
}
img, err := e.getPrebuiltImage()
assert.NoError(t, err)
require.NotNil(t, img)
assert.Equal(t, "helper-image", img.ID)
}
func (e *executor) setPolicyMode(pullPolicy common.DockerPullPolicy) {
e.Config = common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
......
......@@ -428,6 +428,8 @@ func (s *executor) setupBuildPod() error {
}
buildImage := s.Build.GetAllVariables().ExpandValue(s.options.Image.Name)
helperImage := common.AppVersion.Variables().ExpandValue(s.Config.Kubernetes.GetHelperImage())
pod, err := s.kubeClient.CoreV1().Pods(s.configurationOverwrites.namespace).Create(&api.Pod{
ObjectMeta: metav1.ObjectMeta{
GenerateName: s.Build.ProjectUniqueName(),
......@@ -443,7 +445,7 @@ func (s *executor) setupBuildPod() error {
Containers: append([]api.Container{
// TODO use the build and helper template here
s.buildContainer("build", buildImage, s.options.Image, s.buildRequests, s.buildLimits, s.BuildShell.DockerCommand...),
s.buildContainer("helper", s.Config.Kubernetes.GetHelperImage(), common.Image{}, s.helperRequests, s.helperLimits, s.BuildShell.DockerCommand...),
s.buildContainer("helper", helperImage, common.Image{}, s.helperRequests, s.helperLimits, s.BuildShell.DockerCommand...),
}, services...),
TerminationGracePeriodSeconds: &s.Config.Kubernetes.TerminationGracePeriodSeconds,
ImagePullSecrets: imagePullSecrets,
......
......@@ -998,18 +998,51 @@ func TestSetupCredentials(t *testing.T) {
}
}
type setupBuildPodTestDef struct {
RunnerConfig common.RunnerConfig
Variables []common.JobVariable
Options *kubernetesOptions
PrepareFn func(*testing.T, setupBuildPodTestDef, *executor)
VerifyFn func(*testing.T, setupBuildPodTestDef, *api.Pod)
}
type setupBuildPodFakeRoundTripper struct {
t *testing.T
test setupBuildPodTestDef
executed bool
}
func (rt *setupBuildPodFakeRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
rt.executed = true
podBytes, err := ioutil.ReadAll(req.Body)
if !assert.NoError(rt.t, err, "failed to read request body") {
return nil, err
}
p := new(api.Pod)
err = json.Unmarshal(podBytes, p)
if !assert.NoError(rt.t, err, "failed to read request body") {
return nil, err
}
rt.test.VerifyFn(rt.t, rt.test, p)
resp := &http.Response{
StatusCode: http.StatusOK,
Body: FakeReadCloser{
Reader: bytes.NewBuffer(podBytes),
},
}
resp.Header = make(http.Header)
resp.Header.Add("Content-Type", "application/json")
return resp, nil
}
func TestSetupBuildPod(t *testing.T) {
version, _ := testVersionAndCodec()
type testDef struct {
RunnerConfig common.RunnerConfig
Options *kubernetesOptions
PrepareFn func(*testing.T, testDef, *executor)
VerifyFn func(*testing.T, testDef, *api.Pod)
Variables []common.JobVariable
}
tests := []testDef{
{
tests := map[string]setupBuildPodTestDef{
"passes node selector setting": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1021,11 +1054,11 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
assert.Equal(t, test.RunnerConfig.RunnerSettings.Kubernetes.NodeSelector, pod.Spec.NodeSelector)
},
},
{
"uses configured credentials": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1033,19 +1066,19 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
PrepareFn: func(t *testing.T, test testDef, e *executor) {
PrepareFn: func(t *testing.T, test setupBuildPodTestDef, e *executor) {
e.credentials = &api.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: "job-credentials",
},
}
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
secrets := []api.LocalObjectReference{{Name: "job-credentials"}}
assert.Equal(t, secrets, pod.Spec.ImagePullSecrets)
},
},
{
"uses configured image pull secrets": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1056,12 +1089,12 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
secrets := []api.LocalObjectReference{{Name: "docker-registry-credentials"}}
assert.Equal(t, secrets, pod.Spec.ImagePullSecrets)
},
},
{
"configures helper container": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1069,7 +1102,7 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
hasHelper := false
for _, c := range pod.Spec.Containers {
if c.Name == "helper" {
......@@ -1079,7 +1112,7 @@ func TestSetupBuildPod(t *testing.T) {
assert.True(t, hasHelper)
},
},
{
"uses configured helper image": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1088,7 +1121,7 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
for _, c := range pod.Spec.Containers {
if c.Name == "helper" {
assert.Equal(t, test.RunnerConfig.RunnerSettings.Kubernetes.HelperImage, c.Image)
......@@ -1096,7 +1129,7 @@ func TestSetupBuildPod(t *testing.T) {
}
},
},
{
"expands variables for pod labels": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1109,7 +1142,7 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
assert.Equal(t, map[string]string{
"test": "label",
"another": "label",
......@@ -1120,7 +1153,7 @@ func TestSetupBuildPod(t *testing.T) {
{Key: "test", Value: "sometestvar"},
},
},
{
"expands variables for pod annotations": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1133,7 +1166,7 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
assert.Equal(t, map[string]string{
"test": "annotation",
"another": "annotation",
......@@ -1144,7 +1177,24 @@ func TestSetupBuildPod(t *testing.T) {
{Key: "test", Value: "sometestvar"},
},
},
{
"expands variables for helper image": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
Namespace: "default",
HelperImage: "custom/helper-image:${CI_RUNNER_REVISION}",
},
},
},
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
for _, c := range pod.Spec.Containers {
if c.Name == "helper" {
assert.Equal(t, "custom/helper-image:HEAD", c.Image)
}
}
},
},
"supports extended docker configuration for image and services": {
RunnerConfig: common.RunnerConfig{
RunnerSettings: common.RunnerSettings{
Kubernetes: &common.KubernetesConfig{
......@@ -1166,7 +1216,7 @@ func TestSetupBuildPod(t *testing.T) {
},
},
},
VerifyFn: func(t *testing.T, test testDef, pod *api.Pod) {
VerifyFn: func(t *testing.T, test setupBuildPodTestDef, pod *api.Pod) {
require.Len(t, pod.Spec.Containers, 3)
assert.Equal(t, "build", pod.Spec.Containers[0].Name)
......@@ -1187,73 +1237,50 @@ func TestSetupBuildPod(t *testing.T) {
},
}
executed := false
fakeClientRoundTripper := func(test testDef) func(req *http.Request) (*http.Response, error) {
return func(req *http.Request) (resp *http.Response, err error) {
executed = true
podBytes, err := ioutil.ReadAll(req.Body)
if err != nil {
t.Errorf("failed to read request body: %s", err.Error())
return
for testName, test := range tests {
t.Run(testName, func(t *testing.T) {
vars := test.Variables
if vars == nil {
vars = []common.JobVariable{}
}
p := new(api.Pod)
err = json.Unmarshal(podBytes, p)
if err != nil {
t.Errorf("error decoding pod: %s", err.Error())
return
options := test.Options
if options == nil {
options = &kubernetesOptions{}
}
test.VerifyFn(t, test, p)
resp = &http.Response{StatusCode: http.StatusOK, Body: FakeReadCloser{
Reader: bytes.NewBuffer(podBytes),
}}
resp.Header = make(http.Header)
resp.Header.Add("Content-Type", "application/json")
return
}
}
for _, test := range tests {
vars := test.Variables
if vars == nil {
vars = []common.JobVariable{}
}
rt := setupBuildPodFakeRoundTripper{
t: t,
test: test,
}
options := test.Options
if options == nil {
options = &kubernetesOptions{}
}
ex := executor{
kubeClient: testKubernetesClient(version, fake.CreateHTTPClient(fakeClientRoundTripper(test))),
options: options,
AbstractExecutor: executors.AbstractExecutor{
Config: test.RunnerConfig,
BuildShell: &common.ShellConfiguration{},
Build: &common.Build{
JobResponse: common.JobResponse{
Variables: vars,
ex := executor{
kubeClient: testKubernetesClient(version, fake.CreateHTTPClient(rt.RoundTrip)),
options: options,
AbstractExecutor: executors.AbstractExecutor{
Config: test.RunnerConfig,
BuildShell: &common.ShellConfiguration{},
Build: &common.Build{
JobResponse: common.JobResponse{
Variables: vars,
},
Runner: &test.RunnerConfig,
},
Runner: &test.RunnerConfig,
},
},
}
}
if test.PrepareFn != nil {
test.PrepareFn(t, test, &ex)
}
if test.PrepareFn != nil {
test.PrepareFn(t, test, &ex)
}
executed = false
err := ex.prepareOverwrites(make(common.JobVariables, 0))
assert.NoError(t, err, "error preparing overwrites: %s")
err = ex.setupBuildPod()
assert.NoError(t, err, "error setting up build pod: %s")
assert.True(t, executed)
err := ex.prepareOverwrites(make(common.JobVariables, 0))
assert.NoError(t, err, "error preparing overwrites")
err = ex.setupBuildPod()
assert.NoError(t, err, "error setting up build pod")
assert.True(t, rt.executed, "RoundTrip for kubernetes client should be executed")
})
}
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment