Introduce Rendering Performance Pipeline with the SiteSpeed

Issue performance#183 (closed)

This MR introduces Rendering Performance Pipeline that will be triggered by the GPT as an optional job against the reference environments.

Changes:

• Adds generate-sitespeed-urls script to generate URLs for the SiteSpeed using new Environment Config files and URLs Config file.
  • We need to introduce Environment Config file to be able to trigger this pipeline from the GPT using the same ENVIRONMENT_NAME variable.
  • Additionally the script helps to reduce the duplication of URLs files by using the same URLs Config file for different environments.
  • Sadly couldn't find a way to pass dynamic URLs list to the SiteSpeed container, looks like it's not supported so we need to generate URL list in advance.
• Adds reports to wiki/slack similar to what is used in GPT.
  • Report uses the same test names that are used in GPT web tests.
  • TRANSFER SIZE will help to evaluate response page size
  • FRONTEND TIME - The time it takes for the browser to parse and create the page. (docs)
  • FCP - First Contentful Paint(FCP) metric shows when the first viable content (in the form of text, images, etc.)
  • RESULT is being calculated by comparing the Speed Index threshold(2s) with the test result Speed Index.
  • Wiki report is being sent to quality/performance/-/wikis/Benchmarks/sitespeed/ENVIRONMENT_NAME, for example: https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/SiteSpeed/10k
  • Report actions happen at the end of the test job.
    • SiteSpeed Pipeline example: https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/pipelines/157734875
    • GPT + SiteSpeed pipeline example: https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/pipelines/162617571

Example report:

TEST                                  | FCP     | SPEED INDEX       | FRONTEND TIME | TRANSFER SIZE | RESULT
--------------------------------------|---------|-------------------|---------------|---------------|-------
web_group                             | 1791 ms | 2049 ms (<2000ms) | 911 ms        | 1215.3 KB     | FAILED
web_project                           | 2789 ms | 3424 ms (<2000ms) | 897 ms        | 1026.2 KB     | FAILED
web_project_branches                  | 1541 ms | 1579 ms (<2000ms) | 693 ms        | 710.6 KB      | PASSED
web_project_commits                   | 1344 ms | 1370 ms (<2000ms) | 984 ms        | 811.3 KB      | PASSED
web_project_file_blame                | 4421 ms | 4440 ms (<2000ms) | 6892 ms       | 1672.8 KB     | FAILED
web_project_file_rendered             | 1011 ms | 8743 ms (<2000ms) | 725 ms        | 1272.8 KB     | FAILED
web_project_file_source               | 1022 ms | 5044 ms (<2000ms) | 740 ms        | 1619.5 KB     | FAILED
web_project_files                     | 1024 ms | 2144 ms (<2000ms) | 830 ms        | 1135.1 KB     | FAILED
web_project_issue                     | 2858 ms | 2922 ms (<2000ms) | 1627 ms       | 1190.9 KB     | FAILED
web_project_issues                    | 2680 ms | 2701 ms (<2000ms) | 857 ms        | 865.3 KB      | FAILED
web_project_merge_request_changes     | 1690 ms | 7657 ms (<2000ms) | 1423 ms       | 2113.2 KB     | FAILED
web_project_merge_request_commits     | 1521 ms | 2801 ms (<2000ms) | 1391 ms       | 1669.3 KB     | FAILED
web_project_merge_request_discussions | 1625 ms | 2337 ms (<2000ms) | 1410 ms       | 1569.9 KB     | FAILED
web_project_merge_requests            | 2155 ms | 2173 ms (<2000ms) | 850 ms        | 832.7 KB      | FAILED
web_project_pipelines                 | 692 ms  | 1869 ms (<2000ms) | 653 ms        | 876.3 KB      | PASSED
web_user                              | 890 ms  | 1777 ms (<2000ms) | 617 ms        | 797.5 KB      | PASSED

Follow-up issues to create

• Explore performance target and look into the current Speed Index target of 2s - performance#288 (moved)
• Add search web pages - performance#286 (moved)
• Explore if there is a way to catch redirects and browser crashes as well as problem with "heavy" blame route https://staging.gitlab.com/gpt/large_projects/gitlabhq1/-/blame/master/fixtures/emojis/index.json -> performance#287 (moved)
• Investigate failed tests and raise issues if needed, explore additional metrics to add
• Compare results when using n1-highcpu-2 and n1-highcpu-4 for CI runners (SiteSpeed mentioned they use c5.large for testing which should be ~=n1-highcpu-2)
• Update Wiki page with the new format to include SiteSpeed tests

Questions/Room for improvements/Caveats - will be moved to separate issues^:

• Not sure if the RESULT column is helpful. Most of the tests will fail with the current 2s threshold, though we can introduce custom threshold for each url in URL Config file.
• Explore if there is a way to catch redirects
• Currently we suppress budget failures since many pages have Speed Index > 2s. And if the threshold failed, report job won't happen since test and report jobs are combined.
• Single test run takes quite a lot of time due to that some pages are fully loaded after 20-30s and we have 3 iterations for each URL. To handle this problem this project job timeout was increased to 3hrs. Example of such "heavy" pages - http://3k.testbed.gitlab.net/gpt/large_projects/gitlabhq1/-/blame/master/fixtures%2femojis%2findex%2ejson
  • Results: [2020-06-18 17:20:52] INFO: VisualMetrics: FirstVisualChange: 6.37s SpeedIndex: 6.37s PerceptualSpeedIndex: 11.25s ContentfulSpeedIndex: 12.25s VisualComplete85: 6.37s LastVisualChange: 195.17s --> LastVisualChange: 195.17s 🙀
  • The original full run failed because of the 1hr timeout
  • Another example: Testing url http://3k.testbed.gitlab.net/gpt/large_projects/gitlabhq1/-/blame/master/fixtures%2femojis%2findex%2ejson iteration 1 [2020-06-18 20:05:32] ERROR: Failed waiting on page to finished loading, timed out after 300000 ms BrowserError: Running page complete check
    • This data is not helpful, so proposing to use smaller files for blame and rendering.
• Didn't add search urls yet