gitlab runner versions after 14.3.2 are very slow

Summary

We had some runners pinned to version v14.3.2 after the latest security release we decided to jump to the latest version. Normal jobs or quick jobs that should take around 30 seconds to complete were taking minutes.

The jobs normally take around 2 minutes stalled in the "skipping Tried multiple versions from the latest to the v14.3.2.

Steps to reproduce

An example of a pipeline

stages:
  - validate

variables:
  TERRAFORM_VERSION_TAG: "1.1.6"

validate:
  stage: validate
  tags:
    - ec2
  cache:
    key: $CI_PIPELINE_ID
    paths:
      - .terraform
  image:
    name: hashicorp/terraform:${TERRAFORM_VERSION_TAG}
    entrypoint:
      - "/bin/ash"
      - "-c"
  script:
    - terraform --version
    - terraform init
    - terraform validate

Actual behavior

The pipeline on the v14.8.1 takes around 4 minutes.

Expected behavior

The expected pipeline time should be less than a minute

Relevant logs and/or screenshots

Environment description

We are running hosted GitLab runners running the docker executor in an ec2 instance.

config.toml contents

oncurrent = 2
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner@ec2"
  url = "https://gitlab.com"
  token = "redacted"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "redacted"
      BucketLocation = "eu-west-2"
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
    pull_policy = ["if-not-present"]
    shm_size = 0

Used GitLab Runner version

root@78fab90c6522:/# gitlab-runner --version
Version:      14.3.2
Git revision: e0218c92
Git branch:   14-3-stable
GO version:   go1.13.8
Built:        2021-09-30T16:10:52+0000
OS/Arch:      linux/amd64