SAST runs out of memory when copying repo
Summary
SAST runs out of memory when copying the repository contents to the Docker container of the analyzers.
Steps to reproduce
- Choose a big repository using a technology/language supported by SAST
- Configure SAST using the default job definition
- Run the pipeline on any branch
Example Project
So far projects where we can spot this issue are closed source projects.
See https://gitlab.zendesk.com/agent/tickets/105786
What is the current bug behavior?
SAST fails with this error:
fatal error: runtime: out of memory
The error originates from tar.go when copying some file to the tar writer.
What is the expected correct behavior?
Obviously SAST should not fail even when the repo is really big.
Relevant logs and/or screenshots
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc4201fe180 sp=0xc4201fe178 pc=0x452710
runtime.mallocgc(0x18a458000, 0x7c9d40, 0xc420026001, 0xc4201fe258)
/usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc4201fe220 sp=0xc4201fe180 pc=0x410112
runtime.makeslice(0x7c9d40, 0x18a458000, 0x18a458000, 0x428744, 0x88a610, 0xc4201fe2c0)
/usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc4201fe250 sp=0xc4201fe220 pc=0x43ee87
bytes.makeSlice(0x18a458000, 0x0, 0x0, 0x0)
/usr/local/go/src/bytes/buffer.go:230 +0x6d fp=0xc4201fe290 sp=0xc4201fe250 pc=0x4c946d
bytes.(*Buffer).grow(0xc420310310, 0x8000, 0x0)
/usr/local/go/src/bytes/buffer.go:144 +0x151 fp=0xc4201fe2e0 sp=0xc4201fe290 pc=0x4c8e21
bytes.(*Buffer).Write(0xc420310310, 0xc44e45e000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
/usr/local/go/src/bytes/buffer.go:174 +0xd9 fp=0xc4201fe310 sp=0xc4201fe2e0 pc=0x4c90f9
archive/tar.(*regFileWriter).Write(0xc44e34aa40, 0xc44e45e000, 0x8000, 0x8000, 0x0, 0x0, 0x8000)
/usr/local/go/src/archive/tar/writer.go:488 +0xe5 fp=0xc4201fe360 sp=0xc4201fe310 pc=0x75f515
archive/tar.(*Writer).Write(0xc420466000, 0xc44e45e000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
/usr/local/go/src/archive/tar/writer.go:426 +0x7c fp=0xc4201fe3c8 sp=0xc4201fe360 pc=0x75f19c
io.copyBuffer(0x8bed00, 0xc420466000, 0x8bf620, 0xc420331490, 0xc44e45e000, 0x8000, 0x8000, 0xc4201fe4a8, 0x18, 0xc4201fe4a0)
/usr/local/go/src/io/io.go:402 +0x240 fp=0xc4201fe438 sp=0xc4201fe3c8 pc=0x46c290
io.Copy(0x8bed00, 0xc420466000, 0x8bf620, 0xc420331490, 0x0, 0x0, 0x1)
/usr/local/go/src/io/io.go:362 +0x5a fp=0xc4201fe498 sp=0xc4201fe438 pc=0x46c00a
main.Tar.func1(0xc44e3acc80, 0x40, 0x8c43a0, 0xc44e3a7ba0, 0x0, 0x0, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/security-products/sast/tar.go:69 +0x338 fp=0xc4201fe560 sp=0xc4201fe498 pc=0x773d98
path/filepath.walk(0xc44e3acc80, 0x40, 0x8c43a0, 0xc44e3a7ba0, 0xc4203392c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:357 +0x402 fp=0xc4201fe638 sp=0xc4201fe560 pc=0x4f82f2
path/filepath.walk(0xc420021fe0, 0x1b, 0x8c43a0, 0xc44e3a69c0, 0xc4203392c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:381 +0x2c2 fp=0xc4201fe710 sp=0xc4201fe638 pc=0x4f81b2
path/filepath.walk(0xc420021da0, 0x18, 0x8c43a0, 0xc44e3a68f0, 0xc4203392c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:381 +0x2c2 fp=0xc4201fe7e8 sp=0xc4201fe710 pc=0x4f81b2
path/filepath.walk(0xc447b8be50, 0xe, 0x8c43a0, 0xc447d260d0, 0xc4203392c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:381 +0x2c2 fp=0xc4201fe8c0 sp=0xc4201fe7e8 pc=0x4f81b2
path/filepath.walk(0xc43a9346a0, 0x9, 0x8c43a0, 0xc43a8f0f70, 0xc4203392c0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:381 +0x2c2 fp=0xc4201fe998 sp=0xc4201fe8c0 pc=0x4f81b2
path/filepath.walk(0x7fffd2b26f6e, 0x5, 0x8c43a0, 0xc4203b0410, 0xc4203392c0, 0x0, 0x50)
/usr/local/go/src/path/filepath/path.go:381 +0x2c2 fp=0xc4201fea70 sp=0xc4201fe998 pc=0x4f81b2
path/filepath.Walk(0x7fffd2b26f6e, 0x5, 0xc4203392c0, 0x70, 0x82bb80)
/usr/local/go/src/path/filepath/path.go:403 +0x106 fp=0xc4201fead0 sp=0xc4201fea70 pc=0x4f8436
main.Tar(0x7fffd2b26f6e, 0x5, 0x86fc8f, 0x8, 0x8bede0, 0xc420310310, 0x1, 0x3e8, 0x3e8, 0x0, ...)
/go/src/gitlab.com/gitlab-org/security-products/sast/tar.go:25 +0x1aa fp=0xc4201feb18 sp=0xc4201fead0 pc=0x77297a
main.Analyzer.Run.func2(0xc420334a00, 0x40, 0xc4202ec000, 0x8c3100)
/go/src/gitlab.com/gitlab-org/security-products/sast/analyzer.go:118 +0xca fp=0xc4201febb0 sp=0xc4201feb18 pc=0x772cca
main.Analyzer.Run(0xc42006a420, 0x5b, 0x7fffd2b26f6e, 0x5, 0xc4202ec000, 0x1, 0x1, 0x3e8, 0x3e8, 0x8c3100, ...)
/go/src/gitlab.com/gitlab-org/security-products/sast/analyzer.go:189 +0x62a fp=0xc4201feee8 sp=0xc4201febb0 pc=0x7712ea
main.Analyze(0x7fffd2b26f6e, 0x5, 0x1, 0xc4200b1600, 0xa, 0x10, 0x1, 0xc4201cbd70, 0x1bf08eb000, 0x45d964b800, ...)
/go/src/gitlab.com/gitlab-org/security-products/sast/analyze.go:136 +0x9c6 fp=0xc4201ff790 sp=0xc4201feee8 pc=0x76f7f6
main.main.func1(0xc4200d6580, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/security-products/sast/main.go:156 +0x4f4 fp=0xc4201ff968 sp=0xc4201ff790 pc=0x773594
github.com/urfave/cli.HandleAction(0x7d60a0, 0x88a668, 0xc4200d6580, 0x0, 0x0)
/go/src/github.com/urfave/cli/app.go:501 +0xc8 fp=0xc4201ff990 sp=0xc4201ff968 pc=0x73a298
github.com/urfave/cli.(*App).Run(0xc4202e6000, 0xc42000a060, 0x2, 0x2, 0x0, 0x0)
/go/src/github.com/urfave/cli/app.go:268 +0x60c fp=0xc4201ffb78 sp=0xc4201ff990 pc=0x7383bc
main.main()
/go/src/gitlab.com/gitlab-org/security-products/sast/main.go:159 +0x9ad fp=0xc4201fff88 sp=0xc4201ffb78 pc=0x77267d
runtime.main()
/usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc4201fffe0 sp=0xc4201fff88 pc=0x42aba2
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4201fffe8 sp=0xc4201fffe0 pc=0x455121
Possible fixes
It should be possible to set a memory limit and flush the memory before reaching this limit. This would result in creating multiple tar archives if needed. Also, we should skip the files that can't fit in memory - these are not likely to be source code but they can somehow be useful to compile the project (like a dynamic library or a JAR file).
A real fix would be not to rely on tar anymore. We could bind mount the repo directory instead of copying it. Here's how to do this:
- run the
sast
image with no argument - propagate `$CI_PROJECT_DIR``
- mount the repo directory to `$CI_PROJECT_DIR``
To illustrate this, the command line would be:
docker run -ti -e CI_PROJECT_DIR=$CI_PROJECT_DIR \
--volume $CI_PROJECT_DIR:$CI_PROJECT_DIR \
--volume /var/run/docker.sock:/var/run/docker.sock \
registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION
Though bind mounting is already supported this is not the default because it requires to change the job definition of SAST.
Right now the workaround is to change the job definition and use bind mount instead of copy:
sast:
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
allow_failure: true
services:
- docker:stable-dind
script:
- export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
- docker run
--env CI_PROJECT_DIR="$CI_PROJECT_DIR"
--env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}"
--volume "$CI_PROJECT_DIR":"$CI_PROJECT_DIR"
--volume /var/run/docker.sock:/var/run/docker.sock
"registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION"
artifacts:
paths: [gl-sast-report.json]
cc @jwoods06 @tpresa @kristmcg @plafoucriere