Auto DevOps: Improve tests beyond integration tests
Problem to solve
When we develop features for Auto DevOps, we currently we have to use package-and-qa
extensively to make sure Auto DevOps is still functioning after every MR.
A sub-problem is that Auto DevOps is implemented in bash scripts nested inside YAML (gitlab-ci.yml) - our team does not have enough experience to unit-test bash functions
Target audience
~Configure team and contributors to Auto DevOps
Proposal
Extracting separate docker image for deployment stage enables us to test this component in isolation from GitLab and improve our unit test coverage. Right now we have very poor coverage for Auto DevOps and particularly much of our deployment stage is entirely untested.
Working iteratively, we can:
- Extract deploy functions wholesale from
Jobs/Deploy.gitlab-ci.yml
into a new docker image. - Add docker based tests for each major area
At a later stage we can refined our tests, and even switch to a different language (Ruby?) if necessary.
Further details
We have done something similar for the build stage of Auto Devops where we have extracted that into its own docker image - https://gitlab.com/gitlab-org/cluster-integration/auto-build-image.
What does success look like, and how can we measure that?
- Reduce time spent running
package-and-qa
multiple times. Once this is achieved, an automated CI pipeline will provide much faster feedback to the engineer. - Increase amount of tests covering different scenarios. Currently
package-and-qa
only tests one very happy path.