Output list of quarantined endpoints within each run results
It would be nice to see a list of quarantined endpoints within each run, even more ideally with a link to a GitLab issue describing the problem. This will really help folks who aren't as familiar with where we open these issues as we are to find info and follow along.
Right now, in order to quarantine a test it's added to a separate subdirectory within the tests directory: https://gitlab.com/gitlab-org/quality/performance/tree/master/k6/tests/quarantined
It would be great if we could leave the files where they are and edit them to add a quarantine tag + link, similar in idea to how the QA tests get quarantined. Even if we move them into a subfolder and don't change that - being able to store an issue link as a variable and access it for the run output would be very useful.
@grantyoung @niskhakova what do you think?
Our results look like this:
Environment: 50k (12.3.5-ee 9dbaa740018)
Scenario: 60s_1000rps
Date: 2019-10-15
Run Time: 2099.64s (Start: 12:23:48 UTC, End: 12:58:47 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_groups_group | 60.0s | 158.55ms | 949.27/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_groups_projects | 60.0s | 163.63ms | 948.63/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_deploy_keys | 60.0s | 34.81ms | 981.57/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_languages | 60.0s | 29.54ms | 984.81/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests | 60.0s | 11375.05ms | 111.23/s (>200.00/s) | 100.00% (>95%) | Failed
api_v4_projects_merge_requests_merge_request | 60.0s | 1318.22ms | 829.07/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_changes | 60.0s | 1332.41ms | 826.82/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_commits | 60.0s | 15054.72ms | 100.77/s (>80.00/s) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_discussions | 60.0s | 2124.62ms | 574.42/s (>480.00/s) | 100.00% (>95%) | Passed
api_v4_projects_project | 60.0s | 1457.64ms | 765.2/s (>800.00/s) | 100.00% (>95%) | Failed
api_v4_projects_project_pipelines | 60.0s | 60.25ms | 969.8/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_project_search_blobs | 60.0s | 138.38ms | 944.65/s (>400.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_branches_branch | 60.0s | 100.28ms | 956.83/s (>400.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits | 60.0s | 57.81ms | 974.01/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha | 60.0s | 53.07ms | 974.45/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha_diff | 60.0s | 55.48ms | 975.28/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha_signature | 60.0s | 45.44ms | 978.85/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_files_file | 60.0s | 79.30ms | 966.61/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_projects_repository_files_file_raw | 60.0s | 85.83ms | 963.28/s (>800.00/s) | 100.00% (>95%) | Passed
api_v4_user | 60.0s | 26.21ms | 986.15/s (>800.00/s) | 100.00% (>95%) | Passed
projects_blob_controller_show_html | 60.0s | 410.11ms | 927.02/s (>800.00/s) | 100.00% (>95%) | PassedMaybe we can add another table at the bottom like this:
QUARANTINED | ISSUE LINK
---------------------------------------------------------|--------------------------------------------------
api_v4_projects_merge_requests | https://gitlab.com/gitlab-org/gitlab/issues/33150
api_v4_projects_merge_requests_merge_request_commits | https://gitlab.com/gitlab-org/gitlab/issues/31321
api_v4_projects_merge_requests_merge_request_discussions | https://gitlab.com/gitlab-org/gitlab/issues/32455
api_v4_projects_project | https://gitlab.com/gitlab-org/gitlab/issues/33926
api_v4_projects_repository_branches | https://gitlab.com/gitlab-org/gitlab/issues/30536