Semgrep-sast fails with stack overflow error
Summary
Semgrep-sast fails with stack overflow error
Steps to reproduce
When we run semgrep-sast on Java projects, we see a stack overflow error.
We have checked out runner config and they have 8.5gb memory capacity, the container for semgrep is not getting close to that capacity.
[FATA] [Semgrep] [2022-05-09T10:51:24Z] ▶ tool notification error: SemgrepError Error while matching: Semgrep encountered an internal error. This may be a stack overflow. Current stack limit is 10485760, try increasing it via `ulimit -s 20971520`.
semgrep-core exit code: -11
semgrep-core command: /usr/local/lib/python3.9/site-packages/semgrep/bin/semgrep-core -json -rules /tmp/tmp9ey67rf1.yaml -j 32 -targets /tmp/tmpnqk0g0p_ -timeout 30 -timeout_threshold 3 -max_memory 0 -json_time -fast
unexpected non-json output while invoking semgrep-core:
--- semgrep-core stdout ---
--- end semgrep-core stdout ---
--- semgrep-core stderr ---
Existing stack limits: Soft: 10485760, Hard: 10485760
Trying to set soft limit to 10485760
Set stack limit to 10485760, 10485760
--- end semgrep-core stderr ---
An error occurred while invoking the Semgrep engine. Please help us fix this by creating an issue at https://github.com/returntocorp/semgrepExample Project
What is the current bug behavior?
Semgrep Sast fails with stack overflow
What is the expected correct behavior?
Semgrep sast can run successfully
Relevant logs and/or screenshots
$ /analyzer run
[INFO] [Semgrep] [2022-05-09T10:50:32Z] ▶ GitLab Semgrep analyzer v2.21.0
[INFO] [Semgrep] [2022-05-09T10:50:32Z] ▶ Detecting project
[INFO] [Semgrep] [2022-05-09T10:50:32Z] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2022-05-09T10:50:32Z] ▶ Running analyzer
[DEBU] [Semgrep] [2022-05-09T10:50:32Z] ▶ /builds/dwp/get-citizen-income-information-service/income/income-service/.gitlab/sast-ruleset.toml not found, ruleset support will be disabled.
[DEBU] [Semgrep] [2022-05-09T10:50:32Z] ▶ /usr/local/bin/semgrep -f /rules -o /builds/dwp/get-citizen-income-information-service/income/income-service/semgrep.sarif --sarif --no-rewrite-rule-ids --strict --disable-version-check --no-git-ignore --exclude *Test*.java --exclude *.js --exclude target --exclude src/test --enable-metrics /builds/dwp/get-citizen-income-information-service/income/income-service
[DEBU] [Semgrep] [2022-05-09T10:51:24Z] ▶ METRICS: Using configs from the Registry (like --config=p/ci) reports pseudonymous rule metrics to semgrep.dev.
To disable Registry rule metrics, use "--metrics=off".
Using configs only from local files (like --config=xyz.yml) does not enable metrics.
More information: https://semgrep.dev/docs/metrics
Scanning across multiple languages:
java | 90 rules × 133 files
js | 13 rules × 2 files
[INFO] [Semgrep] [2022-05-09T10:51:24Z] ▶ Creating report
[DEBU] [Semgrep] [2022-05-09T10:51:24Z] ▶ Converting report with the root path: /builds/dwp/get-citizen-income-information-service/income/income-service
[FATA] [Semgrep] [2022-05-09T10:51:24Z] ▶ tool notification error: SemgrepError Error while matching: Semgrep encountered an internal error. This may be a stack overflow. Current stack limit is 10485760, try increasing it via `ulimit -s 20971520`.
semgrep-core exit code: -11
semgrep-core command: /usr/local/lib/python3.9/site-packages/semgrep/bin/semgrep-core -json -rules /tmp/tmp9ey67rf1.yaml -j 32 -targets /tmp/tmpnqk0g0p_ -timeout 30 -timeout_threshold 3 -max_memory 0 -json_time -fast
unexpected non-json output while invoking semgrep-core:
--- semgrep-core stdout ---
--- end semgrep-core stdout ---
--- semgrep-core stderr ---
Existing stack limits: Soft: 10485760, Hard: 10485760
Trying to set soft limit to 10485760
Set stack limit to 10485760, 10485760
--- end semgrep-core stderr ---
An error occurred while invoking the Semgrep engine. Please help us fix this by creating an issue at https://github.com/returntocorp/semgrepPossible fixes
Allow us to specify ulimit and --max-memory limits for the semgrep runner within the gitlab-ci file as per: https://semgrep.dev/docs/troubleshooting/semgrep/#semgrep-exited-with-code--11-or--9
Edited by Connor Gilbert