Don't hardcode version number in secure analyzer image integration tests
Problem to solve
The image integration tests added to gemnasium
in gitlab-org/security-products/analyzers/gemnasium!194 (merged) include an analyzer version check:
it "outputs analyzer version" do
analyzer_version = "2.29.7"
expect(@output).to match(/GitLab Gemnasium analyzer v#{analyzer_version}/i)
end
The addition of this test now requires that we update the analyzer version in two places when the changelog entry is updated:
- in the
CHANGELOG.md
file - in the
image_spec.rb
file
If we forget to update the image_spec.rb
file, we end up with a failure:
1) running image with no project outputs analyzer version
Failure/Error: expect(@output).to match(/GitLab Gemnasium analyzer v#{analyzer_version}/i)
expected "\e[0;32m[INFO] [Gemnasium] [2021-08-02T04:56:55Z] ▶ GitLab Gemnasium analyzer v2.29.9\r\n\e[0m\e[0;33m[WARN] [Gemnasium] [2021-08-02T04:56:55Z] ▶ No match in /app\r\n\e[0m" to match /GitLab Gemnasium analyzer v2.29.8/i
Diff:
@@ -1,3 +1,5 @@
-/GitLab Gemnasium analyzer v2.29.8/i
+[INFO] [Gemnasium] [2021-08-02T04:56:55Z] ▶ GitLab Gemnasium analyzer v2.29.9
+[WARN] [Gemnasium] [2021-08-02T04:56:55Z] ▶ No match in /app
+
# ./spec/image_spec.rb:21:in `block (3 levels) in <top (required)>'
Implementation plan
In order to avoid needing to remember to update the changelog version in two places, we need to complete the following steps:
-
Figure out the best approach for ensuring that the version returned by the analyzer matches the most recent version in the CHANGELOG.md
file.This should be implemented in the integration-test project using one of the following options:
-
Add a new
changelog_version
method which will extract the most recent changelog version from theCHANGELOG.md
file, similar to how this is done in the Dockerfile (which was added as part of Report changelog version in Go-based analyzers).We can then compare the value returned by this
changelog_version
method against the one returned by the analyzer in the image integration tests for each project.For example:
it "outputs analyzer version" do changelog_version = GitlabSecure::IntegrationTest::Comparable.changelog_version expect(@output).to match(/GitLab Gemnasium analyzer v#{changelog_version}/i) end
See WIP: Add changelog_version method as a starting point for this approach.
-
Add a shared example which extracts the
changelog_version
and compares it to the version returned by the analyzer. This would simplify and reduce duplication from the image integration tests in the analyzer projects since they can just reference this shared example which contains both the logic for extracting and comparing the version values.For example:
it_behaves_like "analyzer version matches changelog version", "GitLab Gemnasium analyzer"
-
-
Update (or replace) the outputs analyzer version
tests from the following projects to use the new code added in step1.
above: -
Add an outputs analyzer version
test, implemented the same as ingemnasium
andgemnasium-python
above to the following project:
User experience goal
Reduce maintenance cost and unnecessary test failures.
What does success look like, and how can we measure that?
Analyzer image integration tests don't hardcode the changelog version in the spec files, instead they use GitlabSecure::IntegrationTest::Comparable.changelog_version
to automatically extract this information.
What is the type of buyer?
GitLab Ultimate Enterprise Applications
Is this a cross-stage feature?
Yes, this will affect all analyzers that make use of the image integration test project