Bug: Spotbugs Analyzer returns exit code 0 when analyzer fails to build
Summary
Our spotbugs
SAST analyzer currently returns an exit code of 0
even when the project fails to build. When a build fails, an exit code of 1 should occur.
Steps to reproduce
Run analyzer against a test project with a broken pom; i.e. https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs/tree/master/test/broken and check $?
This can also be shown by manually executing the integration test ./test.sh
Example Project
https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs/tree/master/test/broken
What is the current bug behavior?
Analyzer exits with 0
when failing to build project:
❯ IMAGE_TAG=analyzer CI_PROJECT_DIR=$PWD ./test.sh
1..3
No match in /tmp/project
ok 1 - Exit with exit status 3 when project not found
Found project in /tmp/project
Found Maven project in /tmp/project directory
Found 1 analyzable projects.
Building Maven project at /tmp/project.
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-parseable POM /tmp/project/pom.xml: only whitespace content allowed before start tag and not B (position: START_DOCUMENT seen B... @1:1) @ line 1, column 1
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project (/tmp/project/pom.xml) has 1 error
1 #!/bin/sh
[ERROR] Non-parseable POM /tmp/project/pom.xml: only whitespace content allowed before start tag and not B (position: START_DOCUMENT seen B... @1:1) @ line 1, column 1 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
Error: Project couldn't be built: exit status 1
Warning: Building failed for /tmp/project. Attempting scan anyway.
No classfiles specified; output will have no warnings
Warning: SpotBugs didn't find any class file to analyze in /tmp/project !
not ok 2 - Exit with exit status 1 when there's something wrong
Failed 1/2 tests
What is the expected correct behavior?
❯ docker run --interactive --tty --rm --volume "$PWD":/tmp/app --env CI_PROJECT_DIR=/tmp/app analyzer /analyzer run
Found 1 analyzable projects.
Building Maven project at /tmp/project.
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-parseable POM /tmp/project/pom.xml: only whitespace content allowed before start tag and not B (position: START_DOCUMENT seen B... @1:1) @ line 1, column 1
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project (/tmp/project/pom.xml) has 1 error
1 #!/bin/sh
[ERROR] Non-parseable POM /tmp/project/pom.xml: only whitespace content allowed before start tag and not B (position: START_DOCUMENT seen B... @1:1) @ line 1, column 1 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
Error: Project couldn't be built: exit status 1
~/code/gl/security_products/tests/c master*
❯ echo $?
1
Relevant logs and/or screenshots
https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/jobs/181319224
Possible fixes
The analyzer should fail if compilation fails (with the special case for Groovy, failing if both compilation fails).
Development log
Status
-
backend MR to change default behavior and add
FAIL_NEVER
flag gitlab-org/security-products/analyzers/spotbugs!7 (merged) - Update
sast
with newFAIL_NEVER
flag gitlab-org/security-products/sast!149 (merged) - Update
gitlab-ce
template to propagateFAIL_NEVER
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30331 - Change
mvn
/mvnw
build method fromcompile
toinstall
to ensure presence of dependencies gitlab-org/security-products/analyzers/spotbugs!9 (merged)
Decisions
- build failures should result in non-zero exist status
- we should retain the previous behavior with the introduction of a new ENV:
FAIL_NEVER
, named to reflect the similar mvn build flag - This feature unveiled an issue in our test project,
java-maven-multimodules
where cross-dependencies are not being properly resolved; which was suppressed with this ~bug, leading to: - We should switch our base project installation method from
mvn compile
tomvn install
to ensure all cross-project dependencies are present.
Edited by Lucas Charles