Skip to content

SBOM based security scanning UATs - GA

Summary

groupcomposition analysis is moving its new analyzer (dependency scanning) and its integration into the Gitlab monolith into Beta maturity. To conclude that this maturity level is reached in good standing we will define and execute user acceptance tests.

Implementation details

This is the list of all the supported projects for the new DS analyezr.

⚠️ Some of these are not currently supported by the Gemnasium analyzer. If that's the case, skip the corresponding scenario and adjust the test outcome accordingly.

  1. Pick a project from this list and add your name in front of it before starting:
    1. ⚠️ @gonzoyumo - csharp-nuget-dotnetcore => Results: #575148 (closed)
    2. ✅ @gonzoyumo - c-conan => Results: #512405 (closed)
    3. conda with conda-lock.yml (⚠️ no test project available)
      • LOW PRIORITY - very low usage
    4. Dart pub with pubspec.lock and pub.graph.json (⚠️ no test project available)
      • LOW PRIORITY - very low usage
    5. ✅ @ifrenkel go-modules
    6. Java ivy with ivy-report.xml (⚠️ no test project available)
      • LOW PRIORITY - very low usage
    7. ✅ @ifrenkel java-maven => Results: #574944 (closed)
    8. ✅ @ifrenkel java-gradle
    9. ✅ @hacks4oats js-npm
    10. ✅ @hacks4oats js-pnpm => Results: #575613 (closed)
    11. ✅ @hacks4oats js-yarn => Results: #575614 (closed)
    12. ⚠️ @albi.yusupova swift-cocoapods-ios (Podfile.lock) => Results: #575362 (comment 2807041556)
      • We can parse and generate SBOM but we have no advisories for cocoapods in GLAD (no vuln scanning capability) and the Dependency List is empty since we do not support cocoapods purl type there for now.
    13. ⚠️ @albi.yusupova php-composer => Results #575451 (closed)
    14. ✅ @albi.yusupova python-pip
    15. ✅ @nilieskou python-pipenv
    16. ✅ @nilieskou python-poetry => Results: #575368 (closed)
    17. ✅ @nilieskou Python - uv with uv.lock (⚠️ no test project available) => Results: #575422 (closed)
    18. ✅ @gonzoyumo - ruby-bundler - Results: #575456 (closed)
      • Tested with Gemfile.lock and gems.locked (same content but different name - no usage found)
    19. ✅ @onaaman Rust - cargo with Cargo.lock (⚠️ no test project available) => Results: #575716 (closed)
    20. ✅ @onaaman scala-sbt => Results: #575928 (closed)
    21. ✅ @gonzoyumo - swift-spm-serverside (Package.resolved) => Results: #575623 (closed)
  2. Fork the project twice onto the dedicated Gitlab group gitlab-org/secure/tests/sbom-based-security-scanning
    1. Call the first fork - <name of project> - Origin (skip if not supported by gemnasium)
    2. Call the second fork - <name of project> - New
  3. Create a child work item on this issue with the name of the project you are testing and the following description (See also the c-conan example):

    • See work item description template

      ## Projects

1. Original - PROJECT_URL
2. New - PROJECT_URL

## Scenarios

### Baseline with Gemnasium

1. Verify that a pipeline ran on the main branch
   1. [ ] Ensure vulnerability report is fully populated
   1. [ ] Ensure dependency list page is fully populated
      - the vulnerability column
      - the location
      - the dependency paths
   3. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated
2. Create a merge request to the default branch with only README.md changes
   1. [ ] Ensure MR security widget is fully populated (with no findings)
   1. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated 
3. Create an additional MR to the default branch with at least one more vulnerability
   1. [ ] Ensure MR security widget is fully populated with new findings
   1. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated

### New DS analyzer

1. Run a pipeline on the main branch
   1. [ ] Ensure vulnerability report is fully populated
   1. [ ] Ensure dependency list page is fully populated
      - the vulnerability column
      - the location
      - the dependency paths
   3. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated
2. Create a merge request to the default branch with only README.md changes
   1. [ ] Ensure MR security widget is fully populated (with no findings)
   1. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated
3. Create an additional MR to the default branch with an additional vulnerable dependency
   1. [ ] Ensure MR security widget is fully populated with new findings
   1. [ ] Ensure the pipeline tabs (Security and Licenses) are fully populated
4. Create an additional MR to the default branch with the DS report comparison tool
   1. [ ] Verify the pipeline runs both Gemnasium and the new DS analyzer
   1. [ ] Verify the `compare_with_gemnasium` job is successful or report error

## Found Issues
  4. Follow the below scenarios and track results and any weirdness in your sub-work item.
    1. Any unexpected result should be reported to DRIs (@gonzoyumo & @ifrenkel)

Scenarios

Creating baseline follow these steps on "Origin" forks (i.e. using Gemnasium):

⚠️ Before running any pipeline, clean up the .gitlab-ci.yml by removing QA and other unecessary content, keeping the include of the stable v1 template:

include:
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml
  1. Verify that a pipeline ran on the main branch
    1. Ensure vulnerability report is fully populated
    2. Ensure dependency list page is fully populated
    3. Ensure the pipeline tabs (Security and Licenses) are fully populated
  2. Create a merge request to the default branch with only README.md changes
    1. Ensure pipeline finishes successfully
    2. Ensure MR security widget is fully populated (with no findings)
    3. Ensure the pipeline tabs (Security and Licenses) are fully populated
  3. Create an additional MR to the default branch with at least one more vulnerability
    1. Ensure MR security widget is fully populated with new findings
    2. Ensure the pipeline tabs (Security and Licenses) are fully populated

Migrating to new analyzer in "New" forks

⚠️ If the project requires build support (java + python), follow these extra steps:

  1. Read the steps defined in the migration guide to migrate the project to use the new analyzer
  2. Take notes of any failed or unexpected migration step

⚠️ the migration guide is being updated in Update DS on SBOM documentation for Limited ava... (!207251 - merged)

⚠️ Before running any pipeline, clean up the .gitlab-ci.yml by removing QA and other unecessary content, keeping the include of the v2 template (and a build job if necessary):

include:
  - template: Jobs/Dependency-Scanning.v2.gitlab-ci.yml
  1. Run a pipeline on the main branch
    1. Ensure vulnerability report is fully populated
    2. Ensure dependency list page is fully populated
    3. Ensure the pipeline tabs (Security and Licenses) are fully populated
  2. Create a merge request to the default branch with only README.md changes
    1. Ensure pipeline finishes successfully
    2. Ensure MR security widget is fully populated (with no findings)
    3. Ensure the pipeline tabs (Security and Licenses) are fully populated
  3. Create an additional MR to the default branch with an additional vulnerable dependency
    1. Ensure MR security widget is fully populated with new findings
    2. Ensure the pipeline tabs (Security and Licenses) are fully populated
  4. Create an additional MR to the default branch with the DS report comparison tool

    1. Update the .gitlab-ci.yml file with the comparison tool template. NB this template uses MR branch, configure any necessary build job accordingly:

      include:
  - remote: https://gitlab.com/gitlab-org/secure/tests/olivier/ds-analyzers-comparison-tool/-/raw/main/compare-ds-reports.gitlab-ci.yml

    2. Verify the pipeline runs both Gemnasium and the new DS analyzer

    3. Verify the compare_with_gemnasium job is successful or report errors

Compare results between "Origin" and "New" and write any found issue. Link your child item in the list of projects above.

Additional ad hoc scenarios

These are specific use cases that don't have to be exhaustively tested againts each supported project type.

MR pipelines

Combining v1 templates with DS v2

SEP

⚠️ Need !193526 (merged) to be merged and deployed

Other

Known problems

1- ⚠️ pipeline security tab is empty (latency issue)

After pipeline completion, the pipeline security tab seems to take a bit of time to be filled, and displays 0 vulnerabilities for a short moment.

After going back and forth between tabs, the vulnerability count might update, but the list is still empty. A page refresh fixes the problem

It is worth following up with Security Insight group on this, but this is not related to the SBOM based scan using API.

2- ❌ Dependency List page doesn't show vulnerabilities

Again, there might be additional latency to have the vulnerability links with dependencies to be created. It could take a while before we can see which dependencies are vulnerable.

In some cases the vulnerabilities are not showing up at all when using the new DS analyzer whereas they do when using Gemnasium. Example: Gemansium, DS analyzer

Sometimes they show partially (at least temporarily) and for these that stay visible, the sorting by severity doesn't seem to work.

It is worth following up with Security Insight group on this.

3- ✅ Duplicate pipelines on the pipeline list of the Merge Request page

With the default rules in place, a branch pipeline is created immediately upon the push of the new commit, then, when the MR is created, an MR pipeline is created. Both shows up in the pipelines list and might sound confusing but is expected with these rules. Pushing further changes to the MR should only trigger MR pipelines.

4- ❌ MR widget counter is incorrect

In multiple test projects we have seen odd count in the MR widget. In these case we've verified the number of vulnerabilities in the generated DS security reports.

Example 1:

Example 2:

5- ❌ Pipeline security tab counter is incorrect

Example:

cc @willmeek

Edited by Olivier Gonzalez