Output list of known issues of test endpoints
Closes #114 (closed), relates to #125 (closed)
Should be merged after !69 (merged)
Follow-up MR to add list of known issues of test endpoints to Test Details wiki page !75 (merged)
Changes:
- Add module to grep known issues from tests
- Output list of issues in results summary
- Clean up scripts and folder structure
TBD
-
Check the list and make sure all issues are covered
Example results summary:
Known issues:
Note that the following endpoints below have known issues. These tests have either been run with a custom lower threshold limit applied or are quarantined until the issue is fixed:
TEST | ISSUE
------------------------------------------------------------|-----------------------------------------------------------------------------------------------------
api_v4_projects_merge_requests.js | https://gitlab.com/gitlab-org/gitlab/issues/33150, https://gitlab.com/gitlab-org/gitlab/issues/30180
api_v4_projects_merge_requests_merge_request.js | https://gitlab.com/gitlab-org/gitlab/issues/30507
api_v4_projects_merge_requests_merge_request_commits.js | https://gitlab.com/gitlab-org/gitlab/issues/31321
api_v4_projects_merge_requests_merge_request_discussions.js | https://gitlab.com/gitlab-org/gitlab/issues/32455
api_v4_projects_project.js | https://gitlab.com/gitlab-org/gitlab/issues/33926
api_v4_projects_project_search_blobs.js | https://gitlab.com/gitlab-org/gitlab/issues/33562
api_v4_projects_repository_branches.js | https://gitlab.com/gitlab-org/gitlab/issues/30536
api_v4_projects_repository_branches_branch.js | https://gitlab.com/gitlab-org/gitlab/issues/30536
web_project_branches.js | https://gitlab.com/gitlab-org/gitlab/issues/30536
web_project_commits.js | https://gitlab.com/gitlab-org/gitlab/issues/31321
Results summary:
Environment: 10k (12.4.1-ee 8953eae82c8)
Scenario: 60s_200rps
Date: 2019-11-05
Run Time: 1021.55s (Start: 16:15:25 UTC, End: 16:32:26 UTC)
NAME | RPS | RPS RESULT | RESPONSE P95 | REQUEST RESULTS | RESULT
---------------------------------------------------------|-------|----------------------|--------------|-----------------|-------
api_v4_groups_group | 200/s | 151.93/s (>160.00/s) | 1051.97ms | 100.00% (>95%) | Failed
api_v4_groups_projects | 200/s | 164.47/s (>160.00/s) | 638.64ms | 100.00% (>95%) | Passed
api_v4_projects_deploy_keys | 200/s | 160.7/s (>160.00/s) | 324.70ms | 100.00% (>95%) | Passed
api_v4_projects_languages | 200/s | 168.93/s (>160.00/s) | 263.42ms | 100.00% (>95%) | Passed
api_v4_projects_merge_requests | 200/s | 45.8/s (>40.00/s) | 4607.09ms | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request | 200/s | 166.43/s (>160.00/s) | 556.85ms | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_changes | 200/s | 160.8/s (>160.00/s) | 556.71ms | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_commits | 200/s | 18.13/s (>16.00/s) | 11266.76ms | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_discussions | 200/s | 92.2/s (>80.00/s) | 2457.01ms | 100.00% (>95%) | Passed
api_v4_projects_project | 200/s | 157.73/s (>160.00/s) | 636.41ms | 100.00% (>95%) | Failed
api_v4_projects_project_pipelines | 200/s | 165.93/s (>160.00/s) | 567.15ms | 100.00% (>95%) | Passed
api_v4_projects_project_search_blobs | 200/s | 157.43/s (>80.00/s) | 691.40ms | 100.00% (>95%) | Passed
api_v4_projects_repository_branches_branch | 200/s | 166.03/s (>64.00/s) | 393.62ms | 100.00% (>95%) | Passed
api_v4_projects_repository_commits | 200/s | 160.63/s (>160.00/s) | 782.12ms | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha | 200/s | 168.0/s (>160.00/s) | 339.68ms | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha_diff | 200/s | 159.63/s (>160.00/s) | 623.96ms | 100.00% (>95%) | Failed
api_v4_projects_repository_commits_sha_signature | 200/s | 167.9/s (>160.00/s) | 296.26ms | 100.00% (>95%) | Passed
api_v4_projects_repository_files_file | 200/s | 159.23/s (>160.00/s) | 916.04ms | 100.00% (>95%) | Failed
api_v4_projects_repository_files_file_raw | 200/s | 159.33/s (>160.00/s) | 739.66ms | 100.00% (>95%) | Failed
api_v4_projects_repository_tree | 200/s | 161.1/s (>160.00/s) | 391.41ms | 100.00% (>95%) | Passed
api_v4_user | 200/s | 168.8/s (>160.00/s) | 261.33ms | 100.00% (>95%) | Passed
git_ls_remote | 200/s | 8.8/s (>16.00/s) | 16864.37ms | 100.00% (>95%) | Failed
git_pull | 20/s | 9.67/s (>16.00/s) | 3435.63ms | 100.00% (>95%) | Failed
web_group | 20/s | 15.0/s (>16.00/s) | 1147.74ms | 100.00% (>95%) | Failed
web_project | 20/s | 7.2/s (>16.00/s) | 2638.30ms | 100.00% (>95%) | Failed
web_project_branches | 20/s | 7.73/s (>9.60/s) | 2446.90ms | 100.00% (>95%) | Failed
web_project_commits | 20/s | 5.37/s (>9.60/s) | 3563.97ms | 100.00% (>95%) | Failed
web_project_merge_requests | 20/s | 11.43/s (>16.00/s) | 1660.31ms | 100.00% (>95%) | Failed
web_project_pipelines | 20/s | 15.1/s (>16.00/s) | 997.39ms | 100.00% (>95%) | Failed
web_project_tree | 20/s | 10.0/s (>16.00/s) | 2092.40ms | 100.00% (>95%) | Failed
web_projects_blob_controller_show | 20/s | 13.8/s (>16.00/s) | 1283.70ms | 100.00% (>95%) | Failed
Edited by Nailia Iskhakova