Handle new Repository Storage config in Data Generator

Related #291 (closed).

The Data Generator should now support 2 ways to update Repo Storage settings:

• (GitLab 13.0 and earlier) - Repo Storage param repository_storages should contain a list of names of enabled storage paths - the "old method".
• (GitLab 13.1 and later) - Repo Storage param repository_storages_weighted should contain a hash of names of enabled storage paths - the "new method".

To determine which repo storage settings the target environment supports, we should check if repository_storages_weighted exists in Settings response and avoid rely on GitLab version. If the new weighted format is supported, we set all specified storages to 100 and restore the original settings after the the tool is finished.

Changes were validated:

• Against 13.0.0 using old repository_storages param:
  • With a single "default" repo storage
  • With multiple repo storages: "default" and "storage2"

Example output

GPT Data Generator v1.0.4 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://docker' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://docker, Version: 13.0.0 6743e883eae
Creating group test
Creating group test/many_groups_and_projects
Updating GitLab Application Repository Storage setting
Updating application settings: {:"repository_storages[]"=>["default", "storage2"]}
Creating 3 groups with name prefix 'gpt-subgroup-' under parent group 'test/many_groups_and_projects'
...
Creating 3 projects each under 3 subgroups with name prefix 'gpt-project-'
.........
<-> Horizontal data: successfully generated after 8 seconds!

| Vertical data: importing large projects for GPT...
Creating group test/large_projects
Checking if project test1 already exists in test/large_projects/test1...
Updating GitLab Application Repository Storage setting
Updating application settings: {:"repository_storages[]"=>["default"]}
Tarball is remote, downloading...
Starting import of Project 'test1' from tarball 'https://gitlab.com/gpt-bot/export-projects/raw/master/small_project_export.tar.gz' under namespace 'test/large_projects' to GitLab environment 'http://docker'

Checking that GitLab environment 'http://docker' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://docker, Version: 13.0.0 6743e883eae
Importing project test1...
Note that this may take some time to upload a file to the target environment.

Project tarball has successfully uploaded and started to be imported with ID '2511'
Waiting until Project '2511' has imported successfully....
Project has successfully imported in 5 seconds:
http://docker/test/large_projects/test1
Checking if project test2 already exists in test/large_projects/test2...
Updating GitLab Application Repository Storage setting
Updating application settings: {:"repository_storages[]"=>["storage2"]}
Starting import of Project 'test2' from tarball 'https://gitlab.com/gpt-bot/export-projects/raw/master/small_project_export.tar.gz' under namespace 'test/large_projects' to GitLab environment 'http://docker'

Checking that GitLab environment 'http://docker' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://docker, Version: 13.0.0 6743e883eae
Importing project test2...
Note that this may take some time to upload a file to the target environment.

Project tarball has successfully uploaded and started to be imported with ID '2512'
Waiting until Project '2512' has imported successfully....
Project has successfully imported in 5 seconds:
http://docker/test/large_projects/test2


| Vertical data: successfully generated after 20 seconds!
█ GPT data generation finished after 20 seconds.
Restoring the original Repository Storage setting in GitLab Application.
Updating application settings: {:"repository_storages[]"=>["default"]}

█ Logs: results/generate-gpt-data_docker_2020-07-23_193509.log

• Against 13.2.0-pre 8170b984763using newrepository_storages_weighted` param:
  • on local machine using docker image with a single "default" repo storage
  • on 10k env with multiple repo storages

Example output

GPT Data Generator v1.0.4 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://10k.testbed.gitlab.net' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://10k.testbed.gitlab.net, Version: 13.2.0-pre 8170b984763
Group test already exists
Group test/many_groups_and_projects already exists
Updating GitLab Application Repository Storage setting
Updating application settings: {"repository_storages_weighted[default]"=>100, "repository_storages_weighted[storage2]"=>100}
Creating 3 groups with name prefix 'gpt-subgroup-' under parent group 'test/many_groups_and_projects'
...
Creating 3 projects each under 3 subgroups with name prefix 'gpt-project-'
.........
<-> Horizontal data: successfully generated after 11 seconds!

| Vertical data: importing large projects for GPT...
Group test/large_projects already exists
Checking if project test1 already exists in test/large_projects/test1...
Project test/large_projects/test1 already exists
Project version number matches version from the Project Config File.
Existing large project test/large_projects/test1 is valid. Skipping project import...
Checking if project test2 already exists in test/large_projects/test2...
Project test/large_projects/test2 already exists
Project version number matches version from the Project Config File.
Existing large project test/large_projects/test2 is valid. Skipping project import...


| Vertical data: successfully generated after 13 seconds!
█ GPT data generation finished after 13 seconds.
Restoring the original Repository Storage setting in GitLab Application.
Updating application settings: {"repository_storages_weighted[default]"=>100, "repository_storages_weighted[storage1]"=>0, "repository_storages_weighted[storage2]"=>100}

█ Logs: results/generate-gpt-data_10k.testbed.gitlab.net_2020-07-23_175614.log

How it was tested:

1. Ran command to generate data in a test group using a small project to import
2. Ran the same command second time to check its idempotency

Follow-up issue

• We should consider that the breaking change with repository_storages_weighted was introduced in %13.1 while the fix will land in %13.2 (?tentatively), so if a customer will use %13.1 or %13.2 (that doesn't have a fix) the GPT data won't be spread evenly across Gitaly nodes. This will lead to incorrect test results. The fix MR gitlab-org/gitlab!36376 (merged) was just merged this week and it's still unclear in which version/revision it will be. We need to create an issue to add a warning and instruct users to either update or import the projects manually (as well as change the setting via UI).