Skip to content

Testing matrix - Releases vs Secure schemas

Problem Statement

As per gitlab-org/gitlab#497405 (comment 2141268233)

fyi @willmeek integration testing GitLab's current minor releases against the latest analyzers might be valuable to catch this kind of errors more proactively

There was an issue whereby a new schema was unusable on a current release.

We should have tests running for current supported GitLab versions and schema versions, as a backward compatibility test.

Further analysis to occur into how this would be achieved - possibly leverage GitLab QA and specify version?

Where would the test live, as part of our E2E suite, or as Secure Test Projects testing? Or an analyzer downstream test?

Objective

We want to prove out that analyzers work for current supported versions of self managed GitLab

Business Impact

Proves out that breaking changes aren't occurring with analyzers against supported versions of self managed

Customer Base

Customers running previous versions of self managed GitLab who use secure analyzers

Tasks

NOTE that during implementation, it was found that E2E tests won't give us coverage on what we want

  • Create a test project that will execute analyzers - Work iteratively, initially target Dependency Scanning as per original problem - project at https://gitlab.com/gitlab-org/quality/ci/secure-testing/secure-matrix-test - as this is decoupled it will allow us to update it without needing a GitLab-QA release
  • Create a GitLab-QA scenario that will kick off this project. It was found that the following work was needed against the Omnibus instance - gitlab-org/gitlab-qa!1402 (merged)
    • Remove the security modal that pops up
    • Add a license. As we are working outside of the RSpec framework, we can no longer use the codebase there for applying a license
    • Import the project
    • Create a runner
    • Check that the pipeline has succeeded

Conclusion

This tooling is NOT an E2E test, but rather a pipeline that leverages a test project to prove out that analyzers (initially Dependency Scanning) function correctly across all versions.

Edited by Will Meek