unable to parse the server version: invalid character '<'

Summary

discovery_client fails at line 415 while trying to read gitlab version number from REST API for v13.4.7-ee.

Steps to reproduce

helm install --namespace example-namespace \
     -f values.yml \
     example-runner \
     gitlab/gitlab-runner 

.gitlab-ci.yml

stages:
    - test

date_check:
    stage: test
    script: 
        - echo -n "hello world on " && date

Actual behavior

Running with gitlab-runner 13.10.0 (54944146)
  on example-runner-gitlab-runner-56c66dc55d-67wt4 CfikxNvG
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: example-namespace
Using Kubernetes executor with image bash ...
Preparing environment
00:01
ERROR: Job failed (system failure): prepare environment: unable to parse the server version: invalid character '<' looking for beginning of value. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Expected behavior

hello world on Thu Apr  1 14:51:26 EDT 2021

Relevant logs and/or screenshots

job log

hartsocks-a01:k8s hartsocks$ kubectl logs example-runner-gitlab-runner-56c66dc55d-67wt4
Registration attempt 1 of 30
Runtime platform                                    arch=amd64 os=linux pid=15 revision=54944146 version=13.10.0
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Registering runner... succeeded                     runner=kfrjCFEY
Merging configuration from template file "/configmaps/config.template.toml" 
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 
Runtime platform                                    arch=amd64 os=linux pid=1 revision=54944146 version=13.10.0
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Configuration loaded                                builds=0
Metrics server listening                            address=:9252 builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
Checking for jobs... received                       job=556 repo_url=https://[REDACTED].vmware.com/hartsocks/simple-pipelines.git runner=CfikxNvG
ERROR: Job failed (system failure): prepare environment: unable to parse the server version: invalid character '<' looking for beginning of value. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information  duration=327.805107ms job=556 project=6102 runner=CfikxNvG
Checking for jobs... received                       job=557 repo_url=https://[REDACTED].vmware.com/hartsocks/simple-pipelines.git runner=CfikxNvG
WARNING: Failed to process runner                   builds=1 error=prepare environment: unable to parse the server version: invalid character '<' looking for beginning of value. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes runner=CfikxNvG
ERROR: Job failed (system failure): prepare environment: unable to parse the server version: invalid character '<' looking for beginning of value. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information  duration=149.101547ms job=557 project=6102 runner=CfikxNvG
WARNING: Failed to process runner                   builds=0 error=prepare environment: unable to parse the server version: invalid character '<' looking for beginning of value. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes runner=CfikxNvG

Environment description

gitlabUrl: https://[REDACTED]/
runnerRegistrationToken: [REDACTED]
serviceAccountName: [REDACTED]
concurrent: 2
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        image = "bash"
        host = "[REDACTED]"
        namespace = "example-namespace"
        service_account = "[REDACTED]"
        cpu_limit = "1"
        memory_limit = "1Gi"
        service_cpu_limit = "1"
        service_memory_limit = "1Gi"
        helper_cpu_limit = "500m"
        helper_memory_limit = "100Mi"
        poll_interval = 5
        poll_timeout = 3600
        dns_policy = "cluster-first"

config.toml contents

Add your configuration here

Used GitLab Runner version

$ kubectl get deployments -o wide
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                     IMAGES                                 SELECTOR
example-runner-gitlab-runner   1/1     1            1           19m   example-runner-gitlab-runner   gitlab/gitlab-runner:alpine-v13.10.0   app=example-runner-gitlab-runner

Possible fixes

... allow NOT discovering the remote version number. I suspect this error is due to a 404 or 403 error where the unauthenticated discovery REST client is not allowed to access the /version end-point. We may try for a work-around ourselves but in this case we should be able to explicitly set the values that discovery is looking for.