Skip to content

New Results Handling and Automated Comparisons

Grant Young requested to merge gy-k6-comparisons into master

Closes #50 (closed)

This MR facilitates the comparison of multiple test results and uploading them to a wiki page, with the goal of being able to compare results from multiple version of GitLab. To achieve this several challenges had to be tackled and, as such, this MR makes several large improvements to the tool along with facilitating the automated comparisons.

MR makes the following changes:

k6 comparisons pipeline
  • Adds a new k6 comparisons pipeline that spins up our custom made GitLab docker images (via new run-gitlab-docker script) of multiple versions and runs our tests against each in parallel. After completion our new wiki script (report-results-wiki) reports on these results (see below)
  • Our custom Images can be found over on the performance-images project. These are built each month after the new release.
  • Parallel pipelines are limited to be hardcoded and requires us to bump the number each month for now until this is improved.
  • README updated with correct results links
run-k6 updates:
  • The script's underlying results output has been rebuilt to be based on JSON for allowing easier parsing by other scripts. Text table summary is still generated at the end as well so no changes appear on the surface but the underlying JSON results file is now much more useful as raw data.
  • Environment, Options and Test files can be passed as just filenames and no longer needing the full path. Test paths can also be passed or even the name of a folder that contains tests.
    • Loads in each of the above first to report any missing errors earlier
    • Script also shows relative paths instead of full paths in output
  • Script now also has a exclude_tests option that will exclude any tests from being run if their name matches the string given
  • table_print output has been refactored and moved to the run_k6 library to now return the table as a string variable instead of printing directly to STDOUT
  • Result file names have been unified onto one naming convention and now includes the environment version if available
  • Test times are now all found in the new results json file
  • Cleaned up options descriptions to match the above
New scripts \ libraries:
  • ci-report-results-wiki - Replacement for update-wiki that parses one or more k6 test results and uploads them to the given wiki page in Markdown format. When there's more than one result the script will add a comparison table at the top of the page. Example page - https://gitlab.com/gitlab-org/quality/performance/wikis/test_page2.
  • ci-report-test-info-wiki - Replacement for collect-test-info script, this script collects all test info and posts to the give wiki page directly instead of file output.
  • ci-report-results-slack - Refactor of report-results-slack. Script now reports with the JSON results file as well as being able to post a snippet if one results file and a message if none or many. Also updated to use gtp_common library where appropriate.
  • run-gitlab-docker - Convenience script for running a GitLab docker and waiting for the application to be ready. Designed to be used with our new Performance Images
  • New library gpt_common - A common library designed to be shared across all scripts as appropriate. Contains one functions for now named make_http_request
Updated scripts \ libraries:
  • import-project - Updated to use gtp_common library where appropriate
Misc Updates
  • Updated main CI pipelines to use new scripts above and latest Docker images
  • Update gemlist with latest gems
  • Various small updates and improvements
Edited by Grant Young

Merge request reports