Jobs are stuck, logs indicate that POST is failing against https://gitlab.com/api/v4/jobs/request

Summary

Jobs are stuck. Logs indicate that POST is failing against https://gitlab.com/api/v4/jobs/request. Issue started happening 3 days ago without actually doing anything.

Steps to reproduce

This is randomly happening by triggering any of our Pipelines.

Relevant logs and/or screenshots

Checking for jobs... received                       job=XXX repo_url=https://gitlab.com/XXX.git runner=XXX
Checking for jobs... received                       job=XXX repo_url=https://gitlab.com/XXX.git runner=XXX
WARNING: Checking for jobs... failed                runner=XXX status=couldn't execute POST against https://gitlab.com/api/v4/jobs/request: Post https://gitlab.com/api/v4/jobs/request: dial tcp: i/o timeout
WARNING: Checking for jobs... failed                runner=XXX status=couldn't execute POST against https://gitlab.com/api/v4/jobs/request: Post https://gitlab.com/api/v4/jobs/request: dial tcp: i/o timeout

Environment description

AWS EC2 Running c5.2xlarge, Gitlab-Runner Version is 12.3
[[runners]]
  name = "AWS Gitlab Runner XXX"
  url = "https://gitlab.com/"
  token = "xxx"
  executor = "shell"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

12.3.0
Edited by Arpee Ong