Introduce image integration tests in Dependency Scanning analyzers

Summary

Introduce integration tests that check the Docker image and CLI of a Dependency Scanning analyzer project.

Compared to the existing job integration tests (AKA "QA jobs"), these new integration tests can be executed out of a CI job. As a consequence, they don't guarantee that the scanning job integrates properly with the CI. In particular, the Docker image might be incompatible in the job definition or with the CI itself.

Image integration tests are preferable to CLI integration tests because they better reproduce the running environment of the scanning job. That said, there might be cases where it's sufficient to test the analyzer CLI, and where executing docker run only slows down the test with no benefit.

Improvements

It becomes easy to add tests that leverage the existing test projects:

  • tests for supported environment variables (though it's not possible to check the default value set in the job definition)
  • tests for when the supported files are moved to a sub-directory or sub-sub-directory
  • tests using a custom vulnerability database, and resulting in a different report

Right now these tests can be implemented as job integration tests, but this involves creating branches in the Secure test projects, and maintaining them.

Also, it becomes possible to test scenarios where the analyzer fails with a specific exit code. At the moment this can't be achieved with the existing job integration tests. See #324963 (closed)

Risks

See Optional: Missing test coverage

Involved components

All Dependency Scanning analyzers.

Optional: Intended side effects

None

Optional: Missing test coverage

There's a possibly a gap between image and CLI integration tests and the corresponding job integration tests.

Possible gaps:

  • The analyzer behaves differently because of the environment variables. This might be because of:
  • The scanning job is skipped but the analyzer CLI runs.
    • rules:exists doesn't match any file of the scanned directory.
      • Filename doesn't match.
      • Filename does match but directory depth doesn't.

Also, CLI and image integration tests need to be updated whenever the job script changes. If they're not updated, they no longer test the command that's being executed on the CI.

See Dependency Scanning CI template

Implementation plan

  • Demonstrate image integration tests in the gemnasium project
    • List job integration tests that could be implemented as image integration tests
    • Port these to image integration tests
    • Optionally, port the other job integration tests, but skip them by default; see testing community contributions
    • Update project dev documentation, and explain how to run image integration tests
  • Take notes on how to port job integration tests, in this issue
  • Communicate with Secure section
    • Share on Slack
    • Share in a weekly meeting for the Secure stage
Edited by Fabien Catteau