Kubernetes pod getting exited with code 137 when no log is outputted for more than 5 minutes

Summary

Using the Kubernetes executor, if a job does not output logs for more than ~5 minutes, this one gets exited. Resulting on a non-failing GitLab CI job and pod containers exiting with a 137 error code.

After investigating for OOM errors and attempting to reproduce the error with idle pods, I excluded the memory pressure path.

Steps to reproduce

.gitlab-ci.yml

image:
  name: node:12.18.2-alpine

# This job will timeout correctly
loop:
  script:
    - while true; do date; sleep 30; done

# This job exits after 5 min
sleep:
  script:
    - sleep infinity

Actual behavior

The job exits after 5 minutes

Expected behavior

The job should be timing out

Relevant logs and/or screenshots

Outputing every 30s	Not outputting

job log

[0KRunning with gitlab-runner 13.2.1 (efa30e33)
[0;m[0K  on gitlab-runner-shared-4xl-gitlab-runner-68df94c77b-5mgjt Ng8qHjt8
[0;msection_start:1596446218:prepare_executor
[0K[0K[36;1mPreparing the "kubernetes" executor[0;m
[0;m[0KUsing Kubernetes namespace: gitlab-runners
[0;m[0KUsing Kubernetes executor with image node:12.18.2-alpine ...
[0;msection_end:1596446218:prepare_executor
[0Ksection_start:1596446218:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mWaiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Waiting for pod gitlab-runners/runner-ng8qhjt8-project-52-concurrent-1qf9th to be running, status is Pending
Running on runner-ng8qhjt8-project-52-concurrent-1qf9th via gitlab-runner-shared-4xl-gitlab-runner-68df94c77b-5mgjt...
section_end:1596446242:prepare_script
[0Ksection_start:1596446242:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in /builds/max/beta/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out f6d6e0f6 as ci_debug...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1596446243:get_sources
[0Ksection_start:1596446243:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[32;1m$ sleep 3600[0;m
section_end:1596446544:step_script
[0K[32;1mJob succeeded
[0;m

Environment description

I am using self-hosted runners with the kubernetes executore deployed with the helm chart (v0.19.1). With self-hosted GitLab setup also deployed with the helm chart (v4.2.1)

Underlying nodes are EC2 instances, with the following specs:

Kernel Version:             5.4.0-1021-aws
OS Image:                   Ubuntu 20.04.1 LTS
Operating System:           linux
Architecture:               amd64
Container Runtime Version:  docker://19.3.12
Kubelet Version:            v1.17.6

config.toml contents

listen_address = ":9252"
concurrent = 100
check_interval = 10
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-shared-4xl-gitlab-runner-68df94c77b-9cz24"
  output_limit = 4096
  request_concurrency = 1
  url = "https://git.acme.io"
  token = "xxxxx
  executor = "kubernetes"
  environment = []
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "docker:stable"
    namespace = "gitlab-runners"
    namespace_overwrite_allowed = ""
    privileged = false
    cpu_limit = "8"
    memory_limit = "16Gi"
    service_cpu_limit = "1"
    service_memory_limit = "1Gi"
    cpu_request = "8"
    memory_request = "16Gi"
    service_cpu_request = "1"
    service_memory_request = "1Gi"
    helper_cpu_request = "100m"
    helper_memory_request = "256Mi"
    pull_policy = "if-not-present"
    helper_image = "gitlab/gitlab-runner-helper:x86_64-33250107"
    poll_timeout = 900
    service_account = "shared"
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.node_selector]
      "acme.io/pricing-model" = "spot"
    [runners.kubernetes.node_tolerations]
      "acme.io/pricing-model=spot" = "NoSchedule"
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Used GitLab Runner version

Possible fixes

I haven't been able to determine the code responsible for this issue.