Support non-root Dockerfiles in SAST integration-test

Problem to solve

Recently, the Go and C downstream tests in Semgrep were replaced by integration-test. This is part of a larger effort to reduce reliance on downstream QA tests across groupstatic analysis (see #336821 (closed) for more context).

During this work, we found that tests against the FIPS variant of the Semgrep image were failing due to permissions issues. In order to work iteratively, it was decided to exclude the FIPS tests for C and Go in the initial MR.

We're running into permissions issues due to the way that integration-test executes, and because FIPS images run under the gitlab user instead of root:

• The integration-test runner is executed as a Docker container. The current working directory of your computer is bind-mounted into it. The container runs under the root user.
• For each test project (C, Go, etc.) the integration-test container uses Docker-in-Docker to run the analyser. It bind mounts the test project that was bind-mounted from your computer, into the analyser at /app so it can be scanned.
• The analyser executes (under the gitlab user) and attempts to write the gl-sast-report.json artefact into the test project. This action fails because the directory is not owned by gitlab (because it was mounted by the integration-test container) which the gitlab user does not have permissions to write into.

Example

As an example, this is the flow when running integration-test against the Semgrep FIPS image locally.

1. I checkout and build the integration-test Dockerfile locally.

git clone git@gitlab.com:gitlab-org/security-products/analyzers/integration-test.git
cd integration-test
docker build -t integration-test-jliu .

2. I checkout and build Semgrep.

git clone git@gitlab.com:gitlab-org/security-products/analyzers/semgrep.git
cd semgrep
docker build -t semgrep:jliu-fips-integration-test .

3. I run the integration-test runner. Two bind-mounts are created, one to pass through the entirety of the semgrep checkout, and another to pass through my Docker daemon socket.

cd semgrep
docker run -it --rm -v "$PWD:$PWD" -w "$PWD" \
  -e TMP_IMAGE=semgrep:jliu-fips-integration-test \
  -v /var/run/docker.sock:/var/run/docker.sock \
  integration-test-jliu rspec

4. Internally, the integration-test runner executes the analyser against the test project fixture. In this case it's testing the C project.

Notice that it bind-mounts a subdirectory of my semgrep checkout into /app of the analyser container. This is the -v /Users/james/code/gl/security-products/analyzers/semgrep/tmp/test-46070/c/running-image-with-test-project-with-c-behaves-like-successful-scan-creates-a-report:/app param.

docker run -t --rm -v /Users/james/code/gl/security-products/analyzers/semgrep/tmp/test-46070/c/running-image-with-test-project-with-c-behaves-like-successful-scan-creates-a-report:/app -w /app --env ANALYZER_INDENT_REPORT="true" --env CI_PROJECT_DIR="/app" semgrep:jliu-fips-integration-test

5. The analyser runs the scan against the C test project at /app and attempts to write the gl-sast-report.json artefact into /app. This action fails because the analyser is running under the gitlab user, but /app is owned by someone else.

Proposal

I think there are two options:

1. Standardise on non-root users for groupstatic analysis analysers, and adjust the integration-test Dockerfile to use the same user. This requires some coordination, but there has been some prior discussions on using non-root users.
2. Adjust the options passed to the Docker command that spins up the analyser to forcefully use the root user. I tested this locally and it works.

Another option is to override the Docker user in the CI config. When running the non-FIPS integration-test, specify no user, but when running the FIPS integration-test, override to use the gitlab user. Unfortunately it isn't possible to specify an alternative Docker user for GitLab CI jobs.

Addendum

I initially thought the /app directory was owned by root, but it's more peculiar than that. If I run integration-test interactively:

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" \
  -e TMP_IMAGE=semgrep:jliu-fips-integration-test \
  -v /var/run/docker.sock:/var/run/docker.sock \
  integration-test-jliu /bin/ash

and then check the permissions on the $PWD bind mount:

/Users/james/code/gl/security-products/analyzers/semgrep # ls -la
total 228
drwxr-xr-x    1 501      dialout       1056 Nov  2 00:02 .

dialout???

If I then run the analyser container within the integration-test container:

/Users/james/code/gl/security-products/analyzers/semgrep # docker run -it --rm -v /Users/james/code/gl/security-products/analyzers/semgrep/tmp/test-46070/c/running-image-with-test-project-wit
h-c-behaves-like-successful-scan-creates-a-report:/app semgrep:jliu-fips-integration-test /bin/bash

and check the permissions:

[gitlab@0966ddd4af4d /]$ ls -la /app
total 220
drwxr-xr-x 1  501 games    224 Nov  2 03:53 .
drwxr-xr-x 1 root root    4096 Nov  2 04:25 ..
-rw-r--r-- 1  501 games    179 Nov  2 03:53 Makefile
-rw------- 1  501 games   3411 Nov  2 03:53 gl-sast-report.json
-rw-r--r-- 1  501 games     72 Nov  2 03:53 hello.c
-rw-r--r-- 1  501 games 198484 Nov  2 03:53 semgrep.sarif
drwxr-xr-x 1  501 games     96 Nov  2 03:53 subdir

what the heck is games?