Auto DevOps: Improve tests beyond integration tests

Problem to solve

When we develop features for Auto DevOps, we currently we have to use package-and-qa extensively to make sure Auto DevOps is still functioning after every MR.

A sub-problem is that Auto DevOps is implemented in bash scripts nested inside YAML (gitlab-ci.yml) - our team does not have enough experience to unit-test bash functions

Target audience

~Configure team and contributors to Auto DevOps

Proposal

Extracting separate docker image for deployment stage enables us to test this component in isolation from GitLab and improve our unit test coverage. Right now we have very poor coverage for Auto DevOps and particularly much of our deployment stage is entirely untested.

Working iteratively, we can:

  1. Extract deploy functions wholesale from Jobs/Deploy.gitlab-ci.yml into a new docker image.
  2. Add docker based tests for each major area

At a later stage we can refined our tests, and even switch to a different language (Ruby?) if necessary.

Further details

We have done something similar for the build stage of Auto Devops where we have extracted that into its own docker image - https://gitlab.com/gitlab-org/cluster-integration/auto-build-image.

What does success look like, and how can we measure that?

  1. Reduce time spent running package-and-qa multiple times. Once this is achieved, an automated CI pipeline will provide much faster feedback to the engineer.
  2. Increase amount of tests covering different scenarios. Currently package-and-qa only tests one very happy path.

Links / references

Edited by Thong Kuah