Skip to content

Add Performance testing to AutoDevOps

With the work going on in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3507, we will be able to display changes in performance directly within the Merge Request.

This is very similar to how CodeQuality was built. The changes in GitLab itself expect a specific job name to contain a specific file, performance.json in this case. To help customers, we will document some example CI jobs that they can use as a starting point.

For example:

- docker pull registry.gitlab.com/joshlambert/gitlab-exporter/sitespeed-gitlab:master
- docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io registry.gitlab.com/joshlambert/gitlab-exporter/sitespeed-gitlab:master --plugins.add /gitlab-exporter --outputFolder . https://gitlab.com/gitlab-org/gitlab-ce/issues/1 https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3507
- mv data/performance.json performance.json

(Note I am working to find a proper home for the container, it is simply the default sitespeed container with an extra JS file included.)

While that documentation work is being done however, we also should ensure we add this to the AutoDevOps template so it is included for easy customer usage and full scope.

I was envisioning the one of the three following paths: starting up a new "performance" dynamic environment, leveraging the existing review apps, or spin up an environment silently during the job itself.

Performance dynamic environments

The flow would be the following:

  1. Start a temporary environment
  2. Kick off the sitespeed tests against it, in the future load tests (not at the same time)
  3. Shut down the temporary environment

Some thoughts on this model:

  • Given that we also want to expand scope to load testing, it would seem beneficial to use the same environment for both tests.
  • Given current CI support, I think this would require three additional stages: deploy performance environment, run performance tests, clean up environment. This is pretty undesirable, and really clutters up the UI.
  • We'd want to pass the CI_ENVIRONMENT_SLUG from the stage where it was started to the stage where the performance tests are run, so we know where the environment can be reached. Unfortunately this variable is not set on jobs that do not deploy the environment, you can fake this out by adding the environment label but it reports that this job deploys and it does not.

Leverage existing review app

This would simply leverage the existing Review apps that we already spin up, simplifying the pipeline significantly.

Some thoughts on this:

  • We currently don't spin up Review apps for the master branch, which would mean we have nothing to compare a branch to defeating much of the purpose.
  • The issue with the CI_ENVIRONMENT_SLUG would remain here, as well as some risk that interaction with the environment by a user or another test could invalidate the results.
  • This is cleaner from an infrastructure perspective, we'd just need to add a job for master.

Single job which deploys, tests, and cleans up

The final option is to build a single job which deploys a new environment, runs the performance test, and then cleans up. Fully self-contained.

Some thoughts:

  • Less re-use of code and more changes required to build a single job to do all of this
  • Environments are not shared, which is a good thing from a test reliability standpoint, but will take longer to complete and potentially use more resources if tests run in parallel

I have the dynamic performance environments working here: https://gitlab.com/joshlambert/performance-devops/tree/master

It needs cleanup but is functional, the additional stages add complexity for sure though.

I think the best solution would be to have the deploy self-contained within the single job. This is the cleanest from a pipeline standpoint, UX, as well as offers isolation between test environments (barring the VM's stepping on each other on a single host).

@bikebilly and @ayufan do you have thoughts here?

Edited by Joshua Lambert