Randomly happening resource leak (CPU and RAM) while building Node.js app
Summary
Occasionally (once every few days, recently more often) any work with Node.js causes a complete saturation of system resources (CPU and RAM).
Whether it's docker build command (building a Node.js application image) or running tests directly on the node image - the result is the same.
Also it doesn't matter whether gitlab-runner is run in Docker or directly on the system.
We're using node:20-bullseye and node:20-alpine .
Operating systems are:
- Company no. 1 - AlmaLinux 9.2 (Kernel: Linux 5.14.0-284.30.1.el9_2.x86_64)
- Company no. 2 - Rocky Linux 8.8 (Kernel: Linux 4.18.0-477.27.1.el8_8.x86_64)
Steps to reproduce
In my opinion, it's hard to reproduce, because it happens randomly. Example gitlab-ci below.
.gitlab-ci.yml
build-docker:
image:
name: docker:latest
pull_policy: if-not-present
stage: docker
before_script:
- echo ">>> Docker login"
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER
script:
- echo ">>> Building image"
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -f ./Dockerfile-nexus .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
when: manual
Actual behavior
High CPU usage and some kind of memory leak. (happening randomly)
Saturation of system resources (CPU and RAM) cause the operating system to stop responding to SSH or KVM.
Expected behavior
Work correctly?
Relevant logs and/or screenshots
Unfourtunately, I cannot provide full logs for the first job due to company policies...
job log
$ docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -f ./Dockerfile-nexus .
#0 building with "default" instance using docker driver
#1 \[internal\] load build definition from Dockerfile-nexus
#1 transferring dockerfile:
#1 transferring dockerfile: 750B done
#1 DONE 0.1s
#2 \[internal\] load .dockerignore
#2 transferring context: 265B done
#2 DONE 0.1s
#3 \[internal\] load metadata for docker.io/library/nginx:alpine-slim
#3 ...
#4 \[internal\] load metadata for docker.io/library/node:20-bullseye
#4 DONE 2.1s
#3 \[internal\] load metadata for docker.io/library/nginx:alpine-slim
#3 DONE 2.2s
#5 \[stage-1 1/4\] FROM docker.io/library/nginx:alpine-slim@sha256:d45f4c998198583de669cd0ea8ab819ab344d57d6bf42b6f121d0ec097a032e8
#5 resolve docker.io/library/nginx:alpine-slim@sha256:d45f4c998198583de669cd0ea8ab819ab344d57d6bf42b6f121d0ec097a032e8 0.0s done
#5 sha256:e3dd8f3abfcc0450c96d61b58c3e05b3b4fdeb0680c155aa3246f9be4b80343a 1.78kB / 1.78kB done
#5 sha256:ab7c6a6862efb5b590cf30c8e775932fea8d636085f035da206fa28a2e859e85 8.53kB / 8.53kB done
#5 sha256:d45f4c998198583de669cd0ea8ab819ab344d57d6bf42b6f121d0ec097a032e8 1.65kB / 1.65kB done
#5 ...
#6 \[internal\] load build context
#6 transferring context: 6.22MB 0.3s done
#6 DONE 0.4s
#7 \[builder 1/7\] FROM docker.io/library/node:20-bullseye@sha256:33e271b0c446d289f329aedafc7f57c41b3be1429956a7d1e174cfce10993eda
#7 resolve docker.io/library/node:20-bullseye@sha256:33e271b0c446d289f329aedafc7f57c41b3be1429956a7d1e174cfce10993eda 0.0s done
#7 sha256:88893710dcd8be69730988c171d4a88ee71f14338b59a9a23a89ff22cecd6559 2.00kB / 2.00kB done
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 0B / 54.60MB 0.5s
#7 sha256:33e271b0c446d289f329aedafc7f57c41b3be1429956a7d1e174cfce10993eda 1.21kB / 1.21kB done
#7 sha256:3fe66cc46b7ca2f70a6f69da0ac25a7efc8afbf405160bb716188ce53c1c31af 7.52kB / 7.52kB done
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 0B / 55.06MB 0.6s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 0B / 15.76MB 0.6s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 1.05MB / 15.76MB 1.3s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 3.15MB / 54.60MB 1.4s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 2.10MB / 15.76MB 1.5s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 15.73MB / 54.60MB 1.7s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 4.19MB / 55.06MB 1.7s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 5.24MB / 15.76MB 1.7s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 30.41MB / 54.60MB 1.9s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 9.44MB / 15.76MB 1.9s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 34.60MB / 54.60MB 2.0s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 40.89MB / 54.60MB 2.1s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 10.49MB / 55.06MB 2.1s
#7 sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 15.76MB / 15.76MB 2.1s done
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 54.53MB / 54.60MB 2.3s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 15.73MB / 55.06MB 2.3s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 0B / 196.88MB 2.3s
#7 sha256:e4e3163270d4b48adbf1a3df937b8349a555fcd999e96f8d25f23403fb6ab64b 0B / 4.20kB 2.4s
#7 sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 54.60MB / 54.60MB 2.3s done
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 25.17MB / 55.06MB 2.5s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 32.51MB / 55.06MB 2.6s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 37.75MB / 55.06MB 2.7s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 17.83MB / 196.88MB 2.7s
#7 sha256:e4e3163270d4b48adbf1a3df937b8349a555fcd999e96f8d25f23403fb6ab64b 4.20kB / 4.20kB 2.6s done
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 0B / 47.84MB 2.7s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 49.28MB / 55.06MB 3.0s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 30.41MB / 196.88MB 3.0s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 55.06MB / 55.06MB 13.8s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 40.89MB / 196.88MB 13.8s
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 0B / 47.84MB 13.8s
#7 sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 55.06MB / 55.06MB 13.8s done
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 4.19MB / 47.84MB 14.1s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 72.35MB / 196.88MB 18.5s
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 33.55MB / 47.84MB 19.4s
#7 sha256:33ee24534572bfb3819eab610167d3bf0f1f1f64b4ff6d5dd66f287030ec7e18 0B / 2.21MB 19.4s
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 37.75MB / 47.84MB 19.5s
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 85.98MB / 196.88MB 19.6s
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 40.65MB / 47.84MB 19.6s
#7 sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 47.84MB / 47.84MB 19.8s done
#7 sha256:33ee24534572bfb3819eab610167d3bf0f1f1f64b4ff6d5dd66f287030ec7e18 1.05MB / 2.21MB 19.8s
#7 extracting sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 100.66MB / 196.88MB 19.9s
#7 sha256:33ee24534572bfb3819eab610167d3bf0f1f1f64b4ff6d5dd66f287030ec7e18 2.21MB / 2.21MB 19.8s done
#7 sha256:b0d9d6e8e60c3168235aa871796299f1945b335a97f2d7c67d119aa646c72699 0B / 451B 19.9s
#7 ...
#5 \[stage-1 1/4\] FROM docker.io/library/nginx:alpine-slim@sha256:d45f4c998198583de669cd0ea8ab819ab344d57d6bf42b6f121d0ec097a032e8
#5 sha256:740091335c742882a72f93c12141a4664bcc078b1fc4369e0eea4ceab7dd93ee 0B / 3.96MB 20.0s
#5 sha256:740091335c742882a72f93c12141a4664bcc078b1fc4369e0eea4ceab7dd93ee 3.96MB / 3.96MB 20.1s
#5 sha256:da9c2e764c5b41e0bbfc9eb73ac5b2b59da2bdf783bd922fc0bd03619eec5377 0B / 624B 20.1s
#5 extracting sha256:740091335c742882a72f93c12141a4664bcc078b1fc4369e0eea4ceab7dd93ee
#5 sha256:740091335c742882a72f93c12141a4664bcc078b1fc4369e0eea4ceab7dd93ee 3.96MB / 3.96MB 20.1s done
#5 sha256:ade17ad21ef4248e95654886a2cb2156dec3e40862e33ecc7d8ef64cee3c8a9b 0B / 958B 20.2s
#5 sha256:da9c2e764c5b41e0bbfc9eb73ac5b2b59da2bdf783bd922fc0bd03619eec5377 624B / 624B 20.2s done
#5 sha256:ade17ad21ef4248e95654886a2cb2156dec3e40862e33ecc7d8ef64cee3c8a9b 958B / 958B 20.4s done
#5 sha256:4e6f462c8a6979f3030c3fb412b6005166b4d9eb30c8ef00d5b5758399d9d5cf 0B / 369B 20.4s
#5 sha256:1324d9977cd29dcc7f79727f27e22624a72074183dff494ca95d1248ae726f77 0B / 1.21kB 20.4s
#5 sha256:4e6f462c8a6979f3030c3fb412b6005166b4d9eb30c8ef00d5b5758399d9d5cf 369B / 369B 20.5s done
#5 sha256:1324d9977cd29dcc7f79727f27e22624a72074183dff494ca95d1248ae726f77 1.21kB / 1.21kB 20.6s done
#5 sha256:1b9b96da2c74ef457a717b7c55c425ba4a57827d50ccec68105b79c0786a54a4 0B / 1.40kB 20.6s
#5 sha256:1b9b96da2c74ef457a717b7c55c425ba4a57827d50ccec68105b79c0786a54a4 1.40kB / 1.40kB 20.7s
#5 sha256:1b9b96da2c74ef457a717b7c55c425ba4a57827d50ccec68105b79c0786a54a4 1.40kB / 1.40kB 20.7s done
#5 extracting sha256:740091335c742882a72f93c12141a4664bcc078b1fc4369e0eea4ceab7dd93ee 1.2s done
#5 extracting sha256:da9c2e764c5b41e0bbfc9eb73ac5b2b59da2bdf783bd922fc0bd03619eec5377
#5 extracting sha256:da9c2e764c5b41e0bbfc9eb73ac5b2b59da2bdf783bd922fc0bd03619eec5377 0.0s done
#5 extracting sha256:ade17ad21ef4248e95654886a2cb2156dec3e40862e33ecc7d8ef64cee3c8a9b
#5 extracting sha256:ade17ad21ef4248e95654886a2cb2156dec3e40862e33ecc7d8ef64cee3c8a9b 0.0s done
#5 extracting sha256:4e6f462c8a6979f3030c3fb412b6005166b4d9eb30c8ef00d5b5758399d9d5cf
#5 extracting sha256:4e6f462c8a6979f3030c3fb412b6005166b4d9eb30c8ef00d5b5758399d9d5cf 0.1s done
#5 extracting sha256:1324d9977cd29dcc7f79727f27e22624a72074183dff494ca95d1248ae726f77 done
#5 extracting sha256:1b9b96da2c74ef457a717b7c55c425ba4a57827d50ccec68105b79c0786a54a4
#5 extracting sha256:1b9b96da2c74ef457a717b7c55c425ba4a57827d50ccec68105b79c0786a54a4 0.1s done
#5 ...
#7 \[builder 1/7\] FROM docker.io/library/node:20-bullseye@sha256:33e271b0c446d289f329aedafc7f57c41b3be1429956a7d1e174cfce10993eda
#7 sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 196.88MB / 196.88MB 23.8s done
#7 extracting sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 9.4s
#7 sha256:b0d9d6e8e60c3168235aa871796299f1945b335a97f2d7c67d119aa646c72699 451B / 451B 20.0s done
#7 extracting sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 84.0s
#7 ...
#5 \[stage-1 1/4\] FROM docker.io/library/nginx:alpine-slim@sha256:d45f4c998198583de669cd0ea8ab819ab344d57d6bf42b6f121d0ec097a032e8
#5 DONE 104.2s
#7 \[builder 1/7\] FROM docker.io/library/node:20-bullseye@sha256:33e271b0c446d289f329aedafc7f57c41b3be1429956a7d1e174cfce10993eda
#7 extracting sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 89.1s
#7 extracting sha256:2f088d622efd8dbaa13d01eafd0aac8f9f33bb335edd3be897ae8059338c7bf7 89.6s done
#7 extracting sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8
#7 extracting sha256:de448b80f06437f3025dcaa9285d40c9c81e4be00df1b04bd5a26fd6b9447fc8 1.3s done
#7 extracting sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b
#7 extracting sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 5.1s
#7 extracting sha256:24e5da9a0e1f93fa4f1a07ca9a691e0d941bab6927f80157ffc14b478815f95b 6.1s done
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 5.1s
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 21.1s
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 26.2s
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 31.3s
#7 extracting sha256:6d61681290435197504d4cdaa3724700bd66544d064d068e837f70e5abede255 32.5s done
#7 extracting sha256:e4e3163270d4b48adbf1a3df937b8349a555fcd999e96f8d25f23403fb6ab64b 0.0s done
#7 extracting sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049
#7 extracting sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 5.1s
#7 extracting sha256:001918e8eb386ee5da39028f9508381b5ce3686baf3e80c10f0056bd93015049 6.2s done
#7 extracting sha256:33ee24534572bfb3819eab610167d3bf0f1f1f64b4ff6d5dd66f287030ec7e18 0.1s
#7 extracting sha256:33ee24534572bfb3819eab610167d3bf0f1f1f64b4ff6d5dd66f287030ec7e18 0.2s done
#7 extracting sha256:b0d9d6e8e60c3168235aa871796299f1945b335a97f2d7c67d119aa646c72699
#7 extracting sha256:b0d9d6e8e60c3168235aa871796299f1945b335a97f2d7c67d119aa646c72699 done
#7 DONE 158.9s
#8 \[builder 2/7\] RUN apt-get update && apt-get install -y git dos2unix
#8 1.111 Get:1 http://deb.debian.org/debian bullseye InRelease \[116 kB\]
#8 1.190 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease \[48.4 kB\]
#8 1.240 Get:3 http://deb.debian.org/debian bullseye-updates InRelease \[44.1 kB\]
#8 1.765 Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages \[8062 kB\]
#8 2.403 Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages \[258 kB\]
#8 2.991 Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages \[17.4 kB\]
#8 4.755 Fetched 8546 kB in 4s (2344 kB/s)
#8 4.755 Reading package lists...
#8 14.72 Reading package lists...
#8 217.3 Building dependency tree...
#8 217.3 Reading state information...
#8 997.0 git is already the newest version (1:2.30.2-1+deb11u2).
#8 997.0 The following NEW packages will be installed:
#8 997.0 dos2unix
#8 1119.8 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
#8 1119.8 Need to get 392 kB of archives.
#8 1119.8 After this operation, 1353 kB of additional disk space will be used.
#8 1119.8 Get:1 http://deb.debian.org/debian bullseye/main amd64 dos2unix amd64 7.4.1-1 \[392 kB\]
#8 1119.8 Fetched 392 kB in 8min 24s (776 B/s)
#8 1119.8 Selecting previously unselected package dos2unix.
(Reading database ... 22793 files and directories currently installed.)
#8 1119.8 Preparing to unpack .../dos2unix_7.4.1-1_amd64.deb ...
#8 1119.8 Unpacking dos2unix (7.4.1-1) ...
#8 1119.8 Setting up dos2unix (7.4.1-1) ...
#8 1119.9 debconf: delaying package configuration, since apt-utils is not installed
#8 DONE 1121.3s
#9 \[builder 3/7\] WORKDIR /src
#9 DONE 0.2s
#10 \[builder 4/7\] COPY . /src
#10 DONE 0.6s
#11 \[builder 5/7\] RUN yarn --network-timeout=200000 install
#11 5.990 yarn install v1.22.19
#11 6.810 \[1/4\] Resolving packages...
#11 9.152 \[2/4\] Fetching packages...
#11 116.3 info There appears to be trouble with your network connection. Retrying...
Environment description
config.toml contents (Company no. 1)
concurrent = 5
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "***CENSORED***"
url = "***CENSORED***"
id = 6
token = "***CENSORED***"
token_obtained_at = 2023-10-24T18:52:33Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.docker]
tls_verify = false
image = "ruby:2.7"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/builds", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
network_mtu = 0
allowed_pull_policies = ["always", "if-not-present", "never"]
config.toml contents (Company no. 2)
concurrent = 2
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "***CENSORED***"
url = "***CENSORED***"
id = 12
token = "***CENSORED***"
token_obtained_at = 2023-06-26T12:18:35Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.docker]
memory = "7G"
cpus = "3.5"
tls_verify = false
image = "docker:19.03.13"
dns = ["10.1.100.10"]
dns_search = [""]
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
Used GitLab Runner version
16.5.0 (happened also on previous versions, >= 16.0)