Skip to content

Fix integration test scripting

Tomasz Maczukin requested to merge fix-integration-test into main

What does this MR do?

Fixes the integration test

Why was this MR needed?

It seems that recently added validation warnings, example config that we have in our integration test and the way how we check if the test works are causing the job to failure.

In a job executed 3 weeks ago, we can see this:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  22s   default-scheduler  Successfully assigned default/integration-test-e23af4fb-gitlab-runner-579d7fbf67-v2dsh to kind-control-plane
  Normal  Pulling    22s   kubelet            Pulling image "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-bleeding"
  Normal  Pulled     14s   kubelet            Successfully pulled image "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-bleeding" in 7.214225121s
  Normal  Created    14s   kubelet            Created container integration-test-e23af4fb-gitlab-runner
  Normal  Started    14s   kubelet            Started container integration-test-e23af4fb-gitlab-runner
+ timeout 60s grep -m1 'Starting multi-runner' /dev/fd/63
++ kubectl logs -f -l release=integration-test-e23af4fb
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0
+ exit_code=0
+ kubectl logs -l release=integration-test-e23af4fb
Runtime platform                                    arch=amd64 os=linux pid=7 revision=cf848a51 version=15.9.0~beta.157.gcf848a51
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
Initializing executor providers                     builds=0
+ exit 0
Running after_script

while in the same job executed for this MR, it's:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  22s   default-scheduler  Successfully assigned default/integration-test-be96de36-gitlab-runner-995b84bc9-tf8km to kind-control-plane
  Normal  Pulling    22s   kubelet            Pulling image "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-bleeding"
  Normal  Pulled     14s   kubelet            Successfully pulled image "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-bleeding" in 7.171491962s
  Normal  Created    14s   kubelet            Created container integration-test-be96de36-gitlab-runner
  Normal  Started    14s   kubelet            Started container integration-test-be96de36-gitlab-runner
+ timeout 60s grep -m1 'Starting multi-runner' /dev/fd/63
++ kubectl logs -f -l release=integration-test-be96de36
Terminated
+ exit_code=143
+ kubectl logs -l release=integration-test-be96de36
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
WARNING: There might be a problem with your config
jsonschema: '/runners/0/kubernetes/host_aliases' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/kubernetes/$ref/properties/host_aliases/type: expected array, but got null 
Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
Initializing executor providers                     builds=0
+ exit 143
Running after_script

One of the steps in the integration tests is:

timeout 60s grep -m1 "Starting multi-runner" <(kubectl logs -f -l "$INTEGRATION_HELM_POD_RELEASE_LABEL")

so we stream the logs of the deployed Runner pod and seek for Starting multi-runner there. Unfortunately, calling kubectl logs with -l flag - which is the selector - causes it to limit the output to last 10 lines. Looking at the output of kubectl logs --help:

Options:
  (...)
  -l, --selector='': Selector (label query) to filter on.
  (...)
  --tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise
10, if a selector is provided.
  (...)

Our recently added config validation have caused this configuration to throw

WARNING: There might be a problem with your config
jsonschema: '/runners/0/kubernetes/host_aliases' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/kubernetes/$ref/properties/host_aliases/type: expected array, but got null

in the output - quickly at the runner start. This added two additional lines, which practically caused the two first lines of the output (with the Starting multi-runner first line of the log) to be shifted out. And with that the integration test fails on grepping.

What's the best way to test this MR?

What are the relevant issue numbers?

Edited by Tomasz Maczukin

Merge request reports