Skip to content

Always generate secure scanning report regardless of success or failure

DO FIRST

Review this - it was broken down but a large amount of time has passed and it needs to be reviewed to make sure it's update to date. In addition you should see if there are pre-requisites we need to do FIRST, if so notify @NicoleSchwartz and assign to her. You may halt breakdown work as soon as a pre-requisite is found.

Problem to solve

A few things:

We can record more data if we always have an artifact, and we can enable more complex exit codes.

Currently, when a secure scan fails for some reason, the analyzer process fails with an exit error code and no report is generated. In order to provide a "status": "failure" value in the report, as discussed in Add status, start_time, end_time to SAST, CS, DS reports, we'd need to change the report generation behaviour so that a report is always generated, regardless of whether the scan was successful or not.

The purpose of this issue is to update all of the necessary secure analyzer projects so that when a scan fails for some reason, a report is still generated, and the "status": "failure" field is included in the resulting report.

This does NOT include work to handle said reports, that is in this issue

Further details

Side note: this could potentially help with recording analytics around failed jobs.

Please see this discussion for more details

Pros

  • allow storing scan records for every scan, not only successful ones, which will provide more accurate metrics.
  • allow bringing scanning errors into the rails application to be displayed in the UI or notifications

Cons:

  • increase artifact storage usage
  • increase DB records in scan table (if we agree to store failed scans too)

Intended users

User experience goal

A user can rely on the status field of a report to determine whether the scan was successful or not.

Implementation plan

  • Update the run command in the common package and command package so that instead of just exiting when an error occurs, it generates a report with "status":"failure" and then exits with an error code.
  • Upgrade the common / command dependency in analyzer projects depending on command, and release versions.
    • SAST
      • security-code-scan
        • MR link goes here
        • Release link goes here
      • flawfinder
        • MR link goes here
        • Release link goes here
      • brakeman
        • MR link goes here
        • Release link goes here
      • eslint
        • MR link goes here
        • Release link goes here
      • spotbugs
        • MR link goes here
        • Release link goes here
      • gosec
        • MR link goes here
        • Release link goes here
      • bandit
        • MR link goes here
        • Release link goes here
      • phpcs-security-audit
        • MR link goes here
        • Release link goes here
      • sobelow
        • MR link goes here
        • Release link goes here
      • pmd-apex
        • MR link goes here
        • Release link goes here
      • kubesec
        • MR link goes here
        • Release link goes here
      • semgrep
        • MR link goes here
        • Release link goes here
    • Dependency Scanning
      • bundler-audit
        • analyzer
          • MR link goes here
          • Release link goes here
        • tests
          • MR link goes here
      • retire.js
        • MR link goes here
        • Release link goes here
      • gemnasium-python
        • analyzer
          • MR link goes here
          • Release link goes here
        • tests
          • MR link goes here
      • gemnasium-maven
        • analyzer
          • MR link goes here
          • Release link goes here
        • tests
          • MR link goes here
  • Upgrade projects that don't depend on common/command
    • SAST
      • secrets
        • MR link goes here
        • Release link goes here
      • nodejs-scan
        • MR link goes here
        • Release link goes here
    • Dependency Scanning
    • Container Scanning
      • klar
        • MR link goes here
        • Release link goes here

Documentation

Update documentation to explain that when a scan fails, a report will be generated with "status": "failure"

Availability & Testing

To be tested as part of QA, when comparing generated reports with expected ones.

What does success look like, and how can we measure that?

All aforementioned analyzers generate a status field in their reports to reflect the success or failure of the scan.

What is the type of buyer?

GitLab Ultimate

Is this a cross-stage feature?

Yes, and this issue covers all analyzer projects except DAST.

Links / references

/cc @gonzoyumo @NicoleSchwartz

Edited by Nick Ilieskou