Capture helper service logs into job/tasks main trace log [docker]

Status

• MR approved
• merging blocked on completion of gitlab!100349 (merged)

What does this MR do?

This MR captures and streams logs from helper service containers to the CI task/job's main trace logs for the docker executor. This functionality is enabled when the variable CI_DEBUG_SERVICES = 1. While the logs are currently written to the jobs trace logs, they could easily be written elsewhere (file, log aggregator service, syslog...) in the future.

This approach relies on the docker.Client.CaptureLogs() API; I also considered ContainerAttach() as it was also suitable. In the end, for this use case, both API's are very similar; both return an FD in the form of an io.Reader() into which the container's stdout and stderr are multiplexed, and on which stdcopy.StdCopy() can be used to read the contents. CaptureLogs() was a tad simpler so I chose it.

Why was this MR needed?

!2119 (merged) asks for service logs to be captured somehow. This is to help debug failing jobs when the failure is (at least in part) caused by behaviour in one of the service container services' (though not necessarily a failure to start said service container). A few possible approaches are mentioned in the issue; this MR takes the most "iteration friendly" (i.e. simplest) approach of copying the service logs inline into the main trace logs, but leaves room for the logs to be written elsewhere (e.g. a file) in the future.

What's the best way to test this MR?

1. BASELINE: Don't specify CI_DEBUG_SERVICES and run a CI job with a service container. The output of the main log trace should be unchanged from main.

2. Set CI_DEBUG_SERVICES to a bogus value. The error message invalid value '<xxx>' for CI_DEBUG_SERVICES variable should appear in the main trace logs.

Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2786325292

3. Set CI_DEBUG_SERVICES = 1 in the CI configuration
   1. register a runner with a docker executor
   2. create a job that includes a service which writes logs (example below)
   3. run the job (using a runner built from this branch)
   4. the service container's logs should appear in the job's main log trace in grey colour, with the container name prefixed to the log lines.

gitlab-ci.yaml

stages:
  - test

variables:
  POSTGRES_PASSWORD: password
  CI_DEBUG_SERVICES: 1

format:
  stage: test
  image:
    name: golang:latest
  services:
    - postgres:latest
  script:
    - go fmt $(go list ./... | grep -v /vendor/)
    - go vet $(go list ./... | grep -v /vendor/)
    - go test -race $(go list ./... | grep -v /vendor/)

Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2841278059

What are the relevant issue numbers?

• #2119 (closed)
• GitLab runner - include logs for services (dock... (#28063 - closed)
• gitlab#375180 (closed) is relevant only in that it's solutions block us from adding access control of job logs when this feature is enabled (as is done currently with CI_DEBUG_TRACE) as requested by @dcouture

Notes:

• Best reviewed commit-at-a-time.
• Integration tests are still in the works and will be added to THIS MR when complete.
• See !3564 (closed) for the kubernetes executor version of this MR.