End-to-end tests for CI
We need to introduce instance feature tests for CI. We need to verify if CI works well, because CI feature heavily depends on many different moving parts, like:
- GitLab API (registering builds, throttling etc.)
- GitLab Runner (we obviously run tests using Runner)
- Sidekiq (we use concurrency and asynchronous processing for pipelines heavily)
- GitLab Workhorse (we depend on Workhorse to process artifacts for example)
Without good end-to-end tests we may miss something really important, like we did while improving https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6650 (this time staging and idea-to-production works saved us from shipping broken CI pipelines).
This is now, fortunately, easily testable with GitLab QA. We would need to create instance test scenario that:
- creates a new project
- spins-up a docker container with GitLab Runner
- adds runner to the project as a specific runner
- clones an empty repository
- adds some files along with
.gitlab-ci.yml
- pushes commit that should trigger valid pipeline
- pipeline should have at least two stages and some real tests (probably using bash would be enough)
- pipeline should also have artifacts in one of the builds
- pipeline running time should be predictable, we should limit execution time in the scenario
- after pushing commit with
.gitlab-ci.yml
wait for pipeline to finish - verify pipeline status just after push (pipeline pending/running)
- verify pipeline status after given period of time when it should already succeed
- verify build trace and artifacts contents