Skip to content

Spotbugs scanner insufficient memory for the Java Runtime Environment

Summary

Customers running Gitlab Runner Docker Container on Ubuntu 20.04 reporting the following errors causing spotbugs-sast jobs to fail where they were previously successful:

[0.001s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /tmp/app/apf-flowable-api/hs_err_pid1xx.log

This error occurs even if using pre-compilation to skip the build stage and scan the .jar directly.

Two customers reported that this problem does not occur when using version 2.28.5 of the spotbugs docker image.

The amazing @sahbabou investigated the changes between 2.28.5 and 2.28.6 while troubleshooting, here's a summary of what she found:

2.28.5 and 2.28.6 is a change in the spotbugs image. It was bumped from 4.3.0 to 4.4.1, which also includes a version change for:

  • java adoptopenjdk-8.0.302+8 adoptopenjdk-11.0.12+7 from jdk-8.0.292+10 and 11.0.11+9
  • gradle 6.9.1 from 6.9
  • grails 4.0.12 from 4.0.10
  • maven 3.8.2 from 3.8.1

Steps to reproduce

I've not been able to reproduce this locally, but these seem to be the commonalities:

  • Gradle project
  • GitLab Docker executor via GitLab Runner docker container
  • Ubuntu 20.04 host OS for GitLab Runner

Example Project

What is the current bug behavior?

spotbugs-sast job fails almost immediately after begining to build or scan the project with the following error:

Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.

What is the expected correct behavior?

spotbugs-sast scans the project and "just works", no error.

Relevant Support tickets

Relevant logs and/or screenshots

Using docker image sha256:fa446475a7bf07da1a2199b40100a3640d5618c30042d81a1d07f19c9edbc381 for registry.gitlab.com/gitlab-org/security-products/analyzers/spotbugs:2 with digest registry.gitlab.com/gitlab-org/security-products/analyzers/spotbugs@sha256:94cb63a70cb46cae7b78250419a235cabb8df0f4d3108e8dd885a9fef3e21465 ...
$ /analyzer run
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ GitLab Find Security Bugs analyzer v2.28.6
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Detecting project
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Found project in /builds/group/subgroup/project
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Running analyzer
[DEBU] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ /bin/bash -c source /root/.bashrc && switch_to java 11
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Found Gradlew project in /builds/group/subgroup/project directory
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Found 1 analyzable projects.
[INFO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Building Gradlew project at /builds/group/subgroup/project.
[DEBU] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ /builds/group/subgroup/project/gradlew build

[0.002s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /builds/group/subgroup/project/hs_err_pid166.log
[ERRO] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ Project couldn't be built: exit status 1
[FATA] [Find Security Bugs] [2021-10-05T18:28:25Z] ▶ exit status 1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Pin old version of Spotbugs analyzer image

Based on this ticket, this would appear to only affect Spotbugs anaylzer image tags 2.28.6 and up.

SAST_ANALYZER_IMAGE_TAG: "2.28.5"

before_script that remove asdf java, installs and uses alpine openjdk-[8,11] package

spotbugs-sast:
  before_script:
    - . ~/.bashrc
    - asdf plugin remove java && apk add openjdk8 # or openjdk11 for java 11
    - JAVA_HOME=/usr/lib/jvm/java-8-openjdk/ && PATH=$PATH:$JAVA_HOME # or java-11-openjdk for java 11
    - java -version
    - which java
Edited by Greg Myers