Custom Docker container labels are not present on service containers

Summary

Users are able to define custom docker labels in the [runners.docker.container_labels] section of the runner config.toml file.
This feature was introduced in !3304 (merged)

When configured this works for the job container, helper container, etc.. However, there is a bug where these labels are not being applied to the service containers (that is, containers defined in the services: section of a CI config). I suspect service "waiter" containers are also affected.

Steps to reproduce

1. Configure custom labels for the docker executor, for example:

    [runners.docker.container_labels]
      "com.example.mylabel" = "foo"

2. Create a job configuration that uses services: for example:

myjob:
  services:
    - docker:19.03.1-dind  # Look for this container
  script:
    - sleep 120

3. Wait for the job and service to start. On the host system where the runner is located, run docker ps to locate the service container
4. run docker container inspect <CONTAINER ID from step 3> and observe that the com.example.mylabel label is missing.

Actual behavior

The custom label(s) are not present on service containers.

Expected behavior

The custom labels(s) should be present on all containers, including service containers.

Environment description

We use the docker executor in the docker runner (gitlab/gitlab-runner) running on AWS ECS.

toml configuration

listen_address = ":80"
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "My Name"
  output_limit = 1048576
  url = "__REDACTED__"
  token = "__REDACTED__"
  executor = "docker"
  clone_url = "__REDACTED__"
  environment = ["DOCKER_DRIVER=overlay2", "DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }", "AWS_SDK_LOAD_CONFIG=true"]
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "__REDACTED__"
      BucketLocation = "__REDACTED__"
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    hostname = "__REDACTED__"
    image = "__REDACTED__"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    shm_size = 0
    [runners.docker.container_labels]
      "com.datadoghq.tags.env" = "redacted"
      "com.datadoghq.tags.service" = "gitlab-runner-preview-jobs"
      "com.datadoghq.tags.version" = "1"

Used GitLab Runner version

Possible fixes

This bug appears to be due to the fact that the configuration for service containers is created in a separate function from other containers, which was not updated in the MR that implemented this feature.

As I understand it, this line needs to be updated to include labels from the toml config, as is done in createContainerConfig.

It seems service waiter container configs may also have this issue. Though, perhaps that is intentional.